FileSet-FTP

[James Wang]

Are there any known issue(s) with inbound fileset-ftp?

Does it ensure that there will be no message truncation occur?

Topic

[Max Drown (Infor)]

No, it does not. But you can use tcl code in the TPS Directory Parse proc to handle that behavior. For example, you could check the last access time of a file before processing it.

-- Max Drown (Infor)

[Russ Ross]

Max:

I sometimes use another technique and ask the source system to create files with “.pending” while writing to the file, and change it to “.ready” when done writing to the file.

Not elegant but very simple and effective, too.

The question asked by James Wang did get me to wondering does Cloverleaf FileSet-FTP know when FTP failed in the middle of a file transfer, resulting in a partial file transfer, and how does Cloverleaf handle that situation?

Since I don’t know how to reliably terminate a FTP in mid-stream on demand, I think it will be suspect for me to make any determination thru trial and testing.

Long ago we wrote homegrown FTP standalone scripts that deal with these types of robustness issues, and came to realize the status code returned by FTP wasn’t of much use to help with these challenges.

One of the simplest ways is to require an EOF record as the last line of the file, which can be anything you negotiate and not the same as an EOF character, and if missing then you can be certain you did NOT received the whole file.

Unfortunately, this is an indirect method to help identify when a FTP truncation in mid-stream occurred, and most but not all vendors will agree to handle this technique of an EOF record.

So to date, I haven’t used any of the FTP/SFTP/FTPS protocols in Cloverleaf until I can get enough time and confidence to understand how Cloverleaf deals with the occasional malfunctions that occur when doing various File transfers.

If you have any comments about how cloverleaf handles various FTP imperfections like I’m talking about, that would help me with my concerns that has kept me bashful to use Cloverleaf for these protocols.

We have over a hundred batch integrations (FTP/SFTP/FTPS) and none of them use Cloverleaf for the file transfer part, but there is a part of me that wants to expand my comfort zone and consider using Cloverleaf for these protocols.

In other words, is Cloverleaf only as robust as the FTP application or does it try to help it along when FTP has a complete or partial failure?

Russ Ross
RussRoss318@gmail.com

[Max Drown (Infor)]

Checking with R&D … will respond assap.

-- Max Drown (Infor)

[Jim Kosloskey]

As I recall the Fileset protocol maintains a temp file which keeps track of where the transfer is. I think if an interruption occurs and a partial file is transferred, the next directory scan the Fileset protocol recognizes the file is still there and using the temp file picks up where it left off.

Anyway that is the way I think it works.

Since we have not been allowed to use the Fileset/FTP protocol and only minimally the Fileset/Local protocol the abvoe is from memory and not recent usage.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Max Drown (Infor)]

From R&D,

CL ftp protocol uses open source cURL libs.

-- Max Drown (Infor)

[Charlie Bursell]

Jim K. is correct.  You can configure Fileset/FTP to create a temp file which is then moved to the real file name when successful.

One caveat.  The temp file, by default, is created in the FTP login directory which is not necessarily the same directory where the real file goes.  If you want it there be sure and provide full path

[Russ Ross]

Charlie, does this robustness of a temp file occur for a Cloverleaf Fileset-FTP/put or a Fileset-FTP/get or both?

I see the configuration for the temp file on the outbound settings making me think this works for a fileset-FTP/put only and not for the fileset-FTP/get.

Thanks for the heads up on the temp file being created in the FTP login home directory because half the time we cd to another directory that delivers the file(s), especially in the case where the same FTP login is used for TEST & PROD and kept separate with a TEST directory and a PROD directory.

Would have to watch out for TEST and PROD not stepping on each others temp file by making them different names for each environment in this situation, or by specifying a full path like you mentioned to have them created in the delivery directory.

We prefer separate FTP logins for TEST and PROD but don’t always get our way.

Russ Ross
RussRoss318@gmail.com

[Charlie Bursell]

The one I am talking about happens just for the put.  If you are worried about getting partial files use the new facilities of fileset directory proc.

For both FTP and local I have often implemented a procedure where I check the size on each scan and compare to size of previous scan.  I figure if the size is the same between two scans it is stable.

This has been made *MUCH* easier with the new facility that allows you to get the full directory listing.  

I used to have to jump through all sorts of hoops before.  For local I would have to read the NetConfig to get full file path then do a file size.

For FTP I did not use the engine at all but used a cURL script.

Much easier now!

[Author]

Replies