Clovertech

fileset-local grabbing partial file.

› Clovertech Forums › Read Only Archives › Cloverleaf › Cloverleaf › fileset-local grabbing partial file.

  • Creator
    Topic
  • May 6, 2013 at 3:30 pm #53641
    Tim Malaney
    Participant

      We recently upgraded to 5.8.5 on a Linux platform, much larger servers. We have a number of batch processes where we have seen the following – we copy files to a directory where a fileset-local thread is set with a read interval of 2, scan interval of 60 and a max messages of 5. The style is set to eof. We are seeing where files are getting trunctated. At the same time we copy the file to the thread directory be copy to an archive directory so we know the files were complete. Anyone see this issue and have suggestions? We had never see this before.

      Thanks

      Tim

    • Creator
      Topic
    Viewing 4 reply threads
    • Author
      Replies
      • May 6, 2013 at 5:16 pm #78402
        David Barr
        Participant

          This can happen if your thread picks up the file before it has been completely written to disk. There are many different ways to prevent this from happening. My preferred method is to have the process that sends the file to the incoming folder use a temporary filename and rename the file when the transfer is complete. You can use a directory parse proc to ignore the files that match the format of the temporary filename.

          Other methods include putting tests in your directory parse proc on the file modification time or using lsof to see if the file is still open by another process. Or you can try to set up a schedule on the fileset-local protocol so that you only check for files at a time that you know will be after the initial file delivery is complete.

        • May 6, 2013 at 5:38 pm #78403
          Rob Lindsey
          Participant

            I’ve run into this in the past.  What I have done is two different ways.  

            1  – The system or process that is transferring the files write the data file to another directory and then rename the file to the directory your READ process is so that you know that you have a full data file.  

            2 – have a local directory parse program get a list of the data files and then check the size and modification time, put that information into a global or file or database and then the next time that it runs compare the sizes and mod times and if the are the same actually read the data file because you now know that the file is complete.

            I prefer option number one because it is easier to control and then there are not any real delays.

            Rob

          • May 7, 2013 at 6:01 pm #78404
            Tim Malaney
            Participant

              Thanks for the suggestions. We have a script that puts the inbound thread on hold while the process is running. I added stopping the fileset-local thread so it will not try to pick up a file if it happens to hit it’s scan interval.

            • May 7, 2013 at 8:15 pm #78405
              Jim Kosloskey
              Participant

                Tim,

                I am not sure ‘Hold’ works for inbound threads I think it only works for outbound threads.

                email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

              • May 7, 2013 at 8:33 pm #78406
                Tim Malaney
                Participant

                  Thanks Jim, I had a poor choice of words. It is an outbound thread that writes to a file that is then moved to a directory read by a second fileset-local thread.

              • Author
                Replies
              Viewing 4 reply threads
              • The forum ‘Cloverleaf’ is closed to new topics and replies.