Help! Upgrade to 5.7 fileset-ftp — no newlines b/w msgs?

[Jennifer Hardesty]

We are in the process of upgrading from 5.4 to 5.7.  I have set up one of our batch billing ftp processes exactly as it is set up in 5.4.  

However, once the messages begin processing, the ones in a route that are checked in a tclproc for the correct length account # all get killed. The messages that go through the route that just has a translator make it to the next thread fine.

When I compare the SMAT files, the 5.7 inbound SMAT has no x0d and no x0dx0a (newline) characters.  I have the inbound style set to “nl” for both 5.4 and 5.7…and these batch billing files worked perfectly fine in Production last night so I know it’s not something wrong with them.

Is there a setting we missed somewhere in 5.7?

Topic

[Sergey Sevastyanov]

Jennifer,

Do you send to the same FTP server from your new CL 5.7?

We switched to 5.7 from 5.4 over a year ago and I don’t remember running into any problems with FTP threads.

But several months ago we also switched to a different FTP server and we had problems related to the way the new server treats end of lines. I couldn’t fine any details of what exactly happened but I do remember that I had to change ASCII mode to Binary in several occasions to preserve end of lines in transferred files. Again, it wasn’t Cloverleaf related problem but rather the new FTP server, which runs a different FTP service (in fact it runs different OS).

[Jennifer Hardesty]

It’s the same servers.  The only change is the upgrade.

[Michael Hertel]

Cloverleaf’s method of using ftp is different between 5.4 and 5.7

They added Tclcurl.

You probably want to start from scratch to create this interface instead of relying on what you had.

We stayed on 5.4.1 for this reason.

[Jennifer Hardesty]

Well, I did delete all of the ftp processes and folders and begin rebuilding them from scratch because we noticed that some of the settings in 5.4 were carried over to 5.7 but weren’t editable or even seen in 5.7 and were causing errors.

However, I’ve read all of the documentation.  It doesn’t appear to require much different set up, certainly nothing that I can figure out that should be stripping out all of the newlines and other hex characters.

[Michael Hertel]

I never trusted the ascii/binary setting on past Cloverleaf versions.

Do the 50/50 rule. If it doesn’t work as ascii, try binary.

Also check any tclprocs that might be compensating.

Obviously the end result is to get it to work.

[Jennifer Hardesty]

Well, that is just bizarre.  I have switched the Data Type on the FTP Options tab from ASCII to Image/Binary.

Here is what showed up in the log — it looks to me like it’s still using ASCII:

Code:

* About to connect() to 172.xx.x.xx port 21 (#0)
*   Trying 172.xx.x.xx… * connected
* Connected to 172.xx.1.16 (172.xx.x.xx) port 21 (#0)
< 220-Microsoft FTP Service
USER xxxxxxxxxxxx
PASS XXXXXXXX
< 230-You are connected to the Maine Medical Center FTP site.
PWD
CWD /impac/TestFiles
EPSV
* Connect data stream passively
PASV
TYPE A
nlst
< 125 Data connection already open; Transfer starting.
* Maxdownload = -1
* Remembering we are in dir "/impac/TestFiles/"
< 226 Transfer complete.
* Connection #0 to host 172.xx.x.xx left intact
[pd  :pdtd:INFO/1:  fr_impac_ft:04/21/2010 09:21:21] Set driver status to PD_STATUS_UP
[tps :tps :INFO/1:  fr_impac_ft:04/21/2010 09:21:21] tds.string = tps_ftp_impac_dft {MSGID message0} {CONTEXT fileset_ibdirparse} {ARGS {}} {MODE run} {VERSION 3.0}
***

Then, the files were picked up, read in properly, translated and written out just like they currently are in 5.4.

While it’s nice that one is working, I don’t know why and I can’t be sure it’ll work that way when we move it into production.  I’m setting up a test for another ftp batch billing site to see if this is a consistent thing, but this feels very wrong.

[Michael Hertel]

I agree.

[Bob Schmid]

Rather than trying to play the shell game….shouldnt lawson support be able to confirm whether this is broken?

[Jennifer Hardesty]

I spoke to tech support today.  They said that the setting should be binary and that it should have always been binary, even in 5.4.  According to tech support, ASCII would have stripped my newlines and binary keeps them, as opposed to the opposite.

That’s fine and dandy but later I got to studying the log and I really, really don’t think Cloverleaf is recognizing the checkboxes or radio buttons in the FTP Options box on that second tab.  I have it set to delay until needed and close after; however, as soon as I start the process, it immediately attempts to ftp login and then maintains that quiet keepalive connection until it’s needed; if the connection is broken, Cloverleaf won’t know it and it will get stuck when the scheduled time kicks off.

Honestly, even on 5.4, we’ve never been able to get those two options to work on inbound or outbound processes.

Plus, the log still shows that the connection is “Type A” though the files I pick up have all of the x0ds and x0dx0as in all of the right places.

If I had a more trustworthy, consistent, ftp protocol, I would use it.

[Michael Hertel]

I couldn’t agree more.

I quit fighting with ftp a long time ago.

It got to the point that I just compensated with tcl to add/remove

the extra cr/lf’s. The bottom line was that I had to get the interface working.

You would think that by now ftp and batch file processing would work flawlessly and naitively. I suppose it’s more fun to get soap and xml working.

[Jim Kosloskey]

Jennifer and Michael,

Cloverleaf 5.6.

For the United Health Care X12 278 project, I needed to FTPS from UHC.

I used the native Cloverleaf protocol and while we were in use (we stopped the project after a couple of months for data issues not FTP issues) I did not have the issues with either Binary mode or ‘Delay Connection until needed’.

The Binary mode was always set properly as I recall.

As far as the ‘Delay Connection  Until Needed’ I think that is only effective when FTPng to a destination not when Getting from a Server.

So when attempting to Get, the Connection is attempted immediately upon startup as I recall and any available files at that time are presented. Once that set is exhausted, then at the next scan interval (or advanced scheduling trigger) the connection and scanning is done again.

While I have yet to do a FTP Put on Cloverleaf 56. I expect the actual connection will be delayed until there is something to put should I have that turned on.

My experience with the FTP protocol on Cloverelaf 5.6 was not problematic.

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

[Author]

Replies