problem with deleting a file after picking it up

Clovertech Forums Read Only Archives Cloverleaf Cloverleaf problem with deleting a file after picking it up

  • Creator
    Topic
  • #51940
    Joseph Paquette
    Participant

      Hello,

          Have a system that will not tolerate a message with an hl7 test time out of sequence.  What I have done is setup a proc to write each message to a file.  Then I have another thread pick up the files and sort them by the test time, and then send the on there way scheduled to do this once a minute.  The writing to a file works perfectly.  I have a read thread setup as fileset local, have my directory and a directory parse tps proc that puts the messages back in order.  Have searched for info on deletion proc

    Viewing 9 reply threads
    • Author
      Replies
      • #72405
        James Cobane
        Participant

          Joseph,

          The normal processing of an inbound fileset-local thread is to automatically delete the file after it has processed, so I don’t think there is a need to do any additional coding on the inbound thread (the one that is reading the newly written files).  Assuming that the files are named sequentially (which it sounds like they are based on the timestamp naming), then the inbound thread should see the file list in the appropriate order and process them; deleting each file as it is processed.

          Jim Cobane

          Henry Ford Health

        • #72406
          Jim Kosloskey
          Participant

            Are you working in two different directories or trying to do all of this in one directory?

            email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

          • #72407
            Joseph Paquette
            Participant

              Hello,

                  Same directory.

              Joe

            • #72408
              Joseph Paquette
              Participant

                Hello Jim,

                    This is sending to c:save on my test system,  also the read directory is picking up from c:save as well.  

                Joe

              • #72409
                Joseph Paquette
                Participant

                  Here are the .err file and the log from the process folder.  I am using the advanced scheduler to read every 2 minutes, so have 0,2,4… up to 58 in the minutes, have everything else as a *.  What it seems to me is happening is after it processes the first file it doesn

                • #72410
                  Jim Kosloskey
                  Participant

                    Joseph,

                    Is this address path correct:

                    c:save/L.CVICU_L.CV04_A.201008092300000

                    It looks to me like you have a mix of DOS and UNIX directory separators.

                    email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

                  • #72411
                    Shelbi Engle
                    Participant

                      Joe,

                      This sounds like the same or similar issue I am having.  Did you ever find a solution?

                      Thanks!

                      Shelbi Engle

                    • #72412
                      Scott Folley
                      Participant

                        What we need here is a quick overview of how the inbound directory parse and the inbound delete parse scripts work:

                        1.  The inbound directory parse script gets as its argument the list of files in the directory.  Whether you process the file or not (and in what order) is determined by the list that is returned in the disposition.  From the directory parse the engine then gets the list and reads each file according to that list and sends the content to the inbound-TPS for processing.

                        2.  The deletion parse gets the list of files that passed the dir-parse.  If they remain in the list when the message is dispositioned with a continue out of the deletion parse then the engine goes out to the local or remote system and deletes the file.

                        The next thing to understand is that the protocol read is handled by the “read interval”, “scan interval”, and “number of messages” parameters.  If the time that it takes to read the files in exceeds the scan interval then it is quite possible that the directory scan will be done on files that are about to be deleted.  This means that the read will fail because the list is populated with files that are no longer there.  It is important to be careful in the tuning of those parameters so that they take into account the lag time in the processing.  Those parameters do wonders to optimize the communications but they can be a double edged sword.

                        Chances are that this is what you are experiencing.

                        One other thing to keep in mind is the method that is being used to read messages.  If you are reading them “newline terminated” then the number of messages read will be based on the number of newlines in the file that you are reading.  If you are reading the messages as “single” or “eof” then the number of messages parameter will truly be the number of files that you are going to read during the read interval.  I generally prefer to use “single” and split into messages within the inbound-TPS but it is dependent on what your needs are.  If you use “newline terminated” then you will also end up with “fileset-” files in your process directory.  These files are used by the engine to keep track of where it is in a file that has been read in “newline terminated”.

                        Hope that helps.

                      • #72413
                        Joseph Paquette
                        Participant

                          Hello,

                              I did get this working, what we had to do is create a tcl proc for the directory parse that used the glob cmd to grab the directory contents.  Once i had the contents in a list, we were able o sort them and send them out.  I had a high volume of records coming through so it appeared that the read interval could only pick up a small amount of files.  Grabbing the directory made it easier to do.  Here is the code i used for the directory parse.  Not sure it

                        • #72414
                          Scott Folley
                          Participant

                            While there is certainly nothing wrong with doing it that way, basically you are “circumventing” what Cloverleaf already does for you.  If you set your Directory entry in the Fileset Options Inbound to C:/save_msgs/ and set your style to “single” then your Directory Parse proc is going to get the glob’d list that you are manually doing as its message.  At that point you can move right to your lsort in order to get the read to process the messages in the order that you want.  A couple of handy things that you might want to try are “file tail”, “file extension”, and “file rootname”.  These commands give you:

                            file tail: returns the filename without the path

                            file extension: returns the extension of the file

                            file rootname: returns the filename without the extension

                            As one additional point of order, you are doing: return “{CONTINUE $mh}” followed by lappend dispList “CONTINUE $mh”

                            The execution of the proc is going to terminate and return as a direct result of the return command and the lappend is never going to be hit.  You can either do the return as you have it or you can do:

                            lappend dispList “CONTINUE $mh”

                            return $dispList

                            I caution you on that mainly to point out that the return command does not set up the return value, it actually does the return and terminates execution of the proc.

                            These are all just points of order, if what you have done is working for you then that is the important part.

                            One other potentially helpful aside is that I believe the filename is placed into the driver control metadata once you get to the Inbound-TPS (this did not used to be the case).  This is helpful if you need to know the inbound file name while processing the message.

                        Viewing 9 reply threads
                        • The forum ‘Cloverleaf’ is closed to new topics and replies.