fileset-local grabbing partial file.

[Tim Malaney]

We recently upgraded to 5.8.5 on a Linux platform, much larger servers. We have a number of batch processes where we have seen the following – we copy files to a directory where a fileset-local thread is set with a read interval of 2, scan interval of 60 and a max messages of 5. The style is set to eof. We are seeing where files are getting trunctated. At the same time we copy the file to the thread directory be copy to an archive directory so we know the files were complete. Anyone see this issue and have suggestions? We had never see this before.

Thanks

Tim

Topic

[David Barr]

This can happen if your thread picks up the file before it has been completely written to disk. There are many different ways to prevent this from happening. My preferred method is to have the process that sends the file to the incoming folder use a temporary filename and rename the file when the transfer is complete. You can use a directory parse proc to ignore the files that match the format of the temporary filename.

Other methods include putting tests in your directory parse proc on the file modification time or using lsof to see if the file is still open by another process. Or you can try to set up a schedule on the fileset-local protocol so that you only check for files at a time that you know will be after the initial file delivery is complete.

[Rob Lindsey]

I’ve run into this in the past.  What I have done is two different ways.  

1  – The system or process that is transferring the files write the data file to another directory and then rename the file to the directory your READ process is so that you know that you have a full data file.  

2 – have a local directory parse program get a list of the data files and then check the size and modification time, put that information into a global or file or database and then the next time that it runs compare the sizes and mod times and if the are the same actually read the data file because you now know that the file is complete.

I prefer option number one because it is easier to control and then there are not any real delays.

Rob

[Tim Malaney]

Thanks for the suggestions. We have a script that puts the inbound thread on hold while the process is running. I added stopping the fileset-local thread so it will not try to pick up a file if it happens to hit it’s scan interval.

[Jim Kosloskey]

Tim,

I am not sure ‘Hold’ works for inbound threads I think it only works for outbound threads.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Tim Malaney]

Thanks Jim, I had a poor choice of words. It is an outbound thread that writes to a file that is then moved to a directory read by a second fileset-local thread.

[Author]

Replies