fileset-FTP losing messages

[Jim Gibson]

We have an outbound connection set to write messages to a file on our FTP server.  I recently noticed that a few of the messages do not end up in the file on the FTP server.

After I noticed this, I began saving the SMAT on the outbound connection and comparing that file on a daily basis to the file we are writing on the FTP server.  I have discovered each day there are a few messages missing.  I cannot see any rhyme or reason as to why these messages are missing.  There are no FTP protocol errors that I can see around the times the messages are sent and they seem to be randomly spread throughout the day timewise.

Since there is no reply message coming back from the FTP server to acknowledge the receipt of the message, how do we know the message was received and written on the FTP server end?  Has anyone experienced this type of problem?

Topic

[Michael Hertel]

What version and O/S are you on?

Do you have a prewrite proc defined?

[Nora Katz]

My problem is not exactly the same, but also uses the ftp-fileset protocol.  The threads I have set up are supposed to process a file via a tcl proc and create a new file in a different directory and remove the old file, but sometimes the processing is not complete.  Only part of the file is processed and the original file is deleted.  When I requeue the file, the whole file is processed without any problems.  The problem is usually caught because these are billing files and they get rejected when they are not complete.  This problem has also been sporadic.

[Jim Gibson]

We are running Cloverleaf version 5.3 on AIX 5.2.  There are no prewrite procs on this connection.

[Michael Hertel]

Try putting this in your Send Fail Procs in the outbound tab.

It takes an argument in the form of:

{NOTIFYLIST “pgmmjh@vmmc.org pgmtlc@vmmc.org“}

Change the email address and see if you get alerted during the day.

Code:

######################################################################
# Name:         ftp_send_fail_argv
# Purpose:              
# UPoC type:    tps
# Args:         tps keyedlist containing the following keys:
#               MODE    run mode (”start”, “run” or “time”)
#               MSGID   message handle
#               ARGS    user-supplied arguments:
#                      
#
# Returns: tps disposition list:
#          
#

proc ftp_send_fail_argv { args } {
   keylget args MODE mode
   keylget args ARGS.NOTIFYLIST notifylist
   global HciSite HciConnName HciSiteDir
   set procname [lindex [info level [info level]] 0]
   set module “$HciSite/$HciConnName/$procname: ”

   set dispList {}

   switch -exact — $mode {
       start {
       # Perform special init functions
       # N.B.: there may or may not be a MSGID key in args
       }

       run {
       keylget args MSGID mh
       set fname [msgmetaget $mh DRIVERCTL]
       exec echo [msgget $mh] | mail -s “$HciSite – FAILURE Report for file [keylget fname FILESET.OBFILE]” $notifylist
       lappend dispList “CONTINUE $mh”
       }

       time {
       # Timer-based processing
       # N.B.: there may or may not be a MSGID key in args
       }

       shutdown {
           # Doing some clean-up work
       }

       default {
           error “$module Unknown mode ‘$mode'”
       }
   }

   return $dispList
}

[/img]

[Jim Gibson]

Thanks, I have added this proc and will see if I get any emails today.

[Jim Gibson]

I added the above proc and waited.  No emails came.  I compared the output file to the SMAT file and 9 out of 321 messages were missing.  Just for kicks I ran these 9 messages through the  route tester and they all showed they would have been sent.

I guess my question now is when using protocol fileset-FTP, since no ACK messages comes back from a recieving application, how does the engine know if the message was successfully written on the other end?

[Michael Hertel]

Ok, now we’re getting serious.

I’d put this proc in for your Send Succeed proc

and then compare the process log to both the smat file and output file.

You’ll probably need to up your process log size.

If you don’t need to see the full message, I’d change the proc to something like:

       echo $module Sucessful Send: [crange [msgget $mh] 0 200]

or

       echo $module Sucessful Send: [lindex [split [msgget $mh] r] 0]

If that doesn’t help, the next thing I would suggest is creating an EO config to enable fset.

Good luck!

-mh

Code:

######################################################################
# Name:         ftp_send_succeed
# Purpose:              
# UPoC type:    tps
# Args:         tps keyedlist containing the following keys:
#               MODE    run mode (”start”, “run” or “time”)
#               MSGID   message handle
#               ARGS    user-supplied arguments:
#                      
#
# Returns: tps disposition list:
#          
#

proc ftp_send_succeed { args } {
   keylget args MODE mode
   global HciSite HciConnName HciSiteDir
   set procname [lindex [info level [info level]] 0]
   set module “$HciSite/$HciConnName/$procname: ”

   set dispList {}

   switch -exact — $mode {
       start {
       # Perform special init functions
       # N.B.: there may or may not be a MSGID key in args
       }

       run {
       keylget args MSGID mh
       echo $module Sucessful Send: [msgget $mh]
       lappend dispList “CONTINUE $mh”
       }

       time {
       # Timer-based processing
       # N.B.: there may or may not be a MSGID key in args
       }

       shutdown {
           # Doing some clean-up work
       }

       default {
           error “$module Unknown mode ‘$mode'”
       }
   }

   return $dispList
}

[Michael Hertel]

Jim,

Did you get this resolved?

[Jim Gibson]

Not yet.  I hope to work on it this week.  I will let you know what happens.  Thanks

[Jim Gibson]

I put this proc in and just as I expected my log showed all the messages were successful.  It matched what was saved in the SMAT file.  The file on the FTP server is the only one missing records.  My original question still stands.  How does the engine know if a record was successfully saved on the ftp machine?  There is no ACK.

[Anonymous]

Here is what we do.

1. put the file in a folder called in_dir in the ftp server.

2. have a cronjob run and move the file to a directory name dir2.

3. receiving system will get the file from the dir2

I have not seen any messages being lost.

Hope this Helps.

-Reggie

[Michael Hertel]

You can turn up the eo config and see what the engine is doing.

Attached is a neutered process log showing a message as it goes through the outbound thread. The portion you are interested in is really at the bottom.

You’ll see it receives a response from the FTP server that says (in my case) 226 Transfer complete.

I believe that is the answer to your question because it then invokes the send ok procs.

You can also create a custom eo config that includes only fset and pd which should give you enough information to track further.

I’m very interested in your results because I can’t believe that Cloverleaf would loose a message. I’m sure I haven’t lost any.

Let me know what you find,

-mh

[Jim Gibson]

I think you are beginning to see the problem I am describing.  I do not think cloverleaf is loosing messages,  I think they are getting lost on the FTP server.  The message you see in the log is the only thing cloverleaf sees from the FTP server, that it got the message.  It doesn’t let cloverleaf know what it did with it like an ACK coming back from an application would.  I believe there can be a problem on the FTP end and cloverleaf will never know about it unless it gets to the point the FTP server does not accept the message.

[Michael Hertel]

Yes, I see what you are saying.

What kind of FTP server are you talking to?

The next step I would suggest is to leave the last proc in send ok.

Create a new eo alias for fset/control/debug/1 only.

Enable that eo alias and see what response you are getting for each missing message from that server.

One last thing, the file that you are looking at from the other host;

is that a virgin file or has it been processed by some program by the time you look at it?

-mh

[Jim Gibson]

I am going to have a talk with the person in charge of this server to get all the information about it.  I will post what I find out.  

The file is still a virgin when I look at it  🙂

Jim

[Jim Gibson]

I ran a test over the past week.  I created a duplicate connection only this one writes to a file on the engine and then I use a CRON to move the entire file at once to the FTP server once a day.  

Results:  Each day the file that was being written to the FTP server a record at a time lost some data while the file written on the engine and then FTP’d all at once, never lost data.

The FTP server software on the FTP side is just what came with MS 2000 lite (must I say more?).  Moral of the story is to save our data on the engine and then FTP the entire file with a CRON if you want it all to get there.

Thanks for everyone’s help.

Jim

[Author]

Replies