FTP – protocol questions

Clovertech Forums Read Only Archives Cloverleaf Cloverleaf FTP – protocol questions

  • Creator
    Topic
  • #51563
    Mike Campbell
    Participant

      Okay…please bear with the longish post, but here is the situation:

      We are needing to develop an FTP process to retrieve files from an external source [in this case Cerner], pull out the comma deliminated files we need to process, xlt those, and archive them on the local Cloverleaf box.  

      We are currently on 5.7, AIX platform.

      I would imaging the Fileset-FTP protocol.  That would mean having a directory parse and delete tps proc.  Does anyone have an example of the parse and delete procs?  Also, any thoughts on dealing with a comma deliminated file?  

      Or has anyone come up with a cleaner way to do this?  I’ve search the different threads but didn’t really find exactly what I was looking for.

      Thanks all.

      Mike Campbell

    Viewing 6 reply threads
    • Author
      Replies
      • #70779
        Jennifer Hardesty
        Participant

          Though I can’t seem to get it to work, TPS Deletion is supposed to be the default.  It should automatically delete the files after processing.

          I don’t know where the original code for this ftp tcl proc came from but I have modified it to make it generic to use it for all of our processes.

          Code:


          ######################################################################
          # Name: tps_ftp_dft
          # Purpose:  select the files on the host system to processes
          # UPoC type: tps
          # Args: tps keyedlist containing the following keys:
          #       MODE    run mode (”start”, “run” or “test”)
          #       MSGID   message handle
          #       ARGS    user-supplied arguments:
          # FTP parameters are passed via the keyword FTPARGS as
          # a list in the following order separated by a , (comma):
          #
          # FPTRN    = The REGEX Pattern the script will use to recognize
          #                               the file to be picked up
          #
          # Returns: tps disposition list:
          #          Returns a list of messages for the fileset
          #   driver to process.
          #

          proc tps_ftp_dft { args } {
          global HciConnName HciRoot HciSite HciSiteDir FTPIBDIR
          keylget args MODE mode               ;# Fetch mode
          keylget args CONTEXT ctx

          global dft_startup
          set Process “[lindex [split [pwd] /] end]”

          set proc_name “tps_ftp_dft”
          set module  “$proc_name”

          switch -exact — $mode {

          start {
                    # Perform special init functions
          # N.B.: there may or may not be a MSGID key in args

          # set flag to indicate process/thread startup.
          set dft_startup “1”
          echo “[clock format [clock seconds] -format “%Y-%m-%d %H:%M:%S”]”
          echo ” PROC: $proc_name CONTEXT: $ctx MODE: $mode ”
                         return “”
          }

          run {
          # ‘run’ mode always has a MSGID; fetch and process it
          keylget args MSGID mh

          # If we have arguments, fetch them
          keylget args ARGS.FTPRN pattern

          echo “[clock format [clock seconds] -format “%Y-%m-%d %H:%M:%S”]”
          echo ” PROC: $proc_name CONTEXT: $ctx MODE: $mode ”

          # Check for correct context
          if {![cequal $ctx “fileset_ibdirparse”]} {
          echo “INFO: $module – ERROR $proc_name used in wrong context – $ctx”
          echo “Context should be fileset_ibdirparse”
          return “{CONTINUE $mh}”
          }

          # check for startup flag and set off if this is the first run after start.
                  if {[cequal $dft_startup “1”]} {
          set dft_startup “0”
                    msgset $mh {}
                 } else {
          # all other actions in run mode occur only during a scheduled run, not startup run.

          set ibSaveDir [file join $HciSiteDir exec processes]
          set ibSaveDir [file join $ibSaveDir $Process InboundSave]

          # We want to archive the ftp’d files, so…
          #  make sure “ibSaveDir exists, if not create it
          if {![file exists $ibSaveDir]} {
          echo ” INFO: $module –  creating InboundSave directory….”
          if {[catch {file mkdir $ibSaveDir} result]} {
          puts “INFO: $module – ERROR: $ctx – Unable to create “$ibSaveDir””
          puts ”  result was: $result”
          } ;#end if
          } ;#end if      

          # In the context the message handed to this proc
          # by the driver is not a data message.  Rather
          # it is a space seperated string of file names
          # from the directory the fileset driver is
          # configured to read.  
          #
          # The list of files to process is accessed
          # in the same way data messages in other contexts
          # are accessed.  The string is split into a list
          # so that a TCL list sort operation may be
          # performed.  The original message is updated
          # with the contents of the new list, and returned.
          #
          # The fileset driver then processes the files in the
          # order they appear in the returned message list.

          set msg_list [msgget $mh]
          set msg_list [split $msg_list]
          set msg_list [lsort -increasing $msg_list]
          set msg_list [lregexp  $msg_list $pattern]
          echo “***n*** [clock format [clock seconds] -format “%m/%d/%y @ %H:%M:%S”] ***n”
          if {[llength $msg_list] != 0} {
          echo “***n*** INFO: $module – Remote files found to process: $msg_list n***”

          } else {
          echo “****************************************************”
          echo “***n*** INFO: $module  – NO FILE FOUND TO PROCESS n***”
          echo “****************************************************”
          }
          msgset $mh $msg_list
          }
          return “{CONTINUE $mh}”
          }

          time {
          # Timer-based processing
          # N.B.: there may or may not be a MSGID key in args
          }

          shutdown {
          echo “***  $proc_name shutting down.”
          }

          default {
          error “Unknown mode ‘$mode’ in $proc_name”
          }
          }
          }

          If someone can explain to me why IBDEL might not work, I’d love to know.

        • #70780
          Jim Kosloskey
          Participant

            Jennifer,

            I use the Fileset/FTP protocol with  a delete proc and without.

            Without a delete proc, any files that are selected for retrieval (as a result of scanning the defined directory then as modified by any Directory Parse procs) are definitely attempted to be deleted (and in most cases are).

            It may be that the system from which you are retrieving the files does not allow modification of the directory or perhaps deletion of the files. Or there could be some restriction on the providing system regarding issuance of the FTP delete command.

            email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

          • #70781
            Jennifer Hardesty
            Participant

              Jim,

              I put “enable all” EO Config logging on so I could check what Cloverleaf does.  I don’t see that Cloverleaf issues an ftp delete command at all, which is why I am perplexed.  I can use an ftp tool to get to the folder and delete the file myself so I know that the login has permissions.

              Then again nothing peculiar surprises me where ftp, Windows 2003 servers, and Cloverleaf on AIX are concerned anymore.

            • #70782
              Jim Kosloskey
              Participant

                Jennifer,

                What release of Cloverleaf? I think the releases with TclCurl do not report as much activity and so maybe the DELE (I think that is the FTP command) is not getting logged.

                I can’t say specifically I have actually FTP’d from a Windows platform via Cloverleaf(R) but the platforms I have FTP’d from the deletions worked.

                email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

              • #70783
                Mike Campbell
                Participant

                  So, is there a pre-build directory parse tcl proc within 5.7 or do we need to build it from scratch?

                • #70784
                  Jim Kosloskey
                  Participant

                    Mike,

                    I think there are some samples provided.

                    However I have procs I use here for parse and another set for deletion.

                    Send me an email if you would like them and I will send them on to you along with the usage documentation.

                    email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

                  • #70785
                    Jennifer Hardesty
                    Participant

                      Jim,

                      I now know what is happening, though I don’t understand why.  Our production area is 5.4 — we are in the process of upgrading the test environment to 5.7 — and the Read Interval is 5 with a Max Messages of 100.  The ftp file in question contains 1000s of pharmacy hl7 billing messages.

                      From the log I was able to ascertain that Cloverleaf pulls all of the messages from the file at one time rather than 100 at intervals, which is what I would expect.  Cloverleaf takes all of the messages from the actual file and writes them to a local semi-temporary file called -FTP-temp — I call it “semi-temporary” because it looks like it should be temporary but it doesn’t get cleaned up by Cloverleaf.  Once all the messages are in the local file, Cloverleaf starts processing them.

                      Now, when I first posted on this thread that the DELE message wasn’t being sent, it turns out that a timeout on the connection between the pharmacy server and Cloverleaf was occurring while Cloverleaf was taking the 30+ minutes to process the 1000s of messages in -FTP-temp.  Therefore, by the time Cloverleaf finished processing the individual messages, there was no connection to send the DELE message to and didn’t bother to nor did it even log an error that it couldn’t.

                      After I requested that the pharmacy increase their keepalive to 900 sec because the timeout was actually causing us issues opening the connection every night when the Advanced Scheduler triggered the file pickup, there was no timeout during the 30+ minutes Cloverleaf took to process the 1000s of messages in -FTP-temp.  Therefore, after Cloverleaf completed processing the individual messages, a DELE was sent to the ftp folder and the file was deleted successfully.

                      So, what I don’t understand is

                    • * Why do the Read Interval and Max Messages fields work on the locally created temp file rather than the actual file in the ftp folder?  What sense does that make?

                      * If Cloverleaf is going to pull all of the messages into a temp file for processing, why does it wait to do IBDEL until after the processing of the messages is complete?  The two functions should be independent.

                    • Thanks in advance!

                  Viewing 6 reply threads
                  • The forum ‘Cloverleaf’ is closed to new topics and replies.