VRL with CR File Processing

[Jon Melin]

Hello,

I have some files that I need to run through an XLATE to create a tab delimited file output. I’ve used the same VRL for the output in the past, and and that functions fine. The issue is the files I’ve received have carriage returns (CR) at the end. I built and inbound VRL to take the files in, and it function fine but only every processes the first line. It only processes the first line when I use EOF (end of file), and grabs the first line. If I try to do New Line separated (which in Notepad ++ it is a new line, but just with CR) it says it reaches the end of the file without a new line.

“20170811.txt — read 2272 characters but no newline”

I know this has something to do with it not seeing the correct CR/LF because when I manually open the file and return at the end and save it, it processes as new line without an issue.

Anyone have any ways of dealing with this? I think I have the correct global properties for the inbound VRL setup, but it still doesn’t see CR as a new line in the engine for some reason. I’m sure I’m missing something super simple, any suggestions? I’ve included a screenshot of some of my configurations and a screenshot that shows the CR at the end.

Please let me know if anyone has any suggestions. Thank you in advance.

Topic

[Jim Kosloskey]

Have you tried ‘single’? I am not sure if that will work but if it does I think  it will treat the entire file as a single message and then your defined terminator may have a chance to work.

Remember the entire file will be treated as a single message if this works. If it is a large file that could pose some capacity/performance issues for you.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jon Melin]

Jim,

I tried single and I still only get the first line and the rest are cut off. I can’t seem to get PWIM to see the (CR) as a reason to move down to the next line, even when I put x0D as the global termination on the VRL config.

Any other thoughts I could try?

Thank you,

Jon

[Jim Kosloskey]

If you could get the sending system to add a NL or replace the CR with a NL then you would be set.

The Global terminator for the VRL only is effective I think inside an HRL definition.

You could use a Tcl proc at the Inbound Tps with single and then inside that Tcl proc create a message for each split on CR or something like that.

I suppose you could also create an HRL (if the VRL has a field you can specify the repeat constraint against) and indicate the VRL repeats – then the terminator would come into play.

Feel free to contact me off line if you want.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Steve Williams]

Jon,

Notepad++ recognizes 3 different file formats with line terminators:

DOS/Windows >> CRNL

Macitosh >> CR

Unix >> NL

When you open a normal HL7 message file, Notepadd++ sees the CR at the end of the “first line” an then assumes the file opened a Mac generated file. Which is good and desirable for us.

For cloverleaf, which is designed to work with Unix and Windows based systems, LF (linefeed = x0A) means either the newline character in Unix, and the CRLF character set means newline in Windows.

Cloverleaf does not see a CR carriage return (x0D) as a “newline”. As such, the VRL testing tool will not function as desired because the tool is looking for a newline terminator when reading the file. Long before the data can be parsed by the actual VRL code.

This same problem occurs with the protocol file reader. It wants a newline terminator to break up a file into “messages,” in this case a series of VRL records where each record is a single data message.

Even if you set the VRL Global properties to use a record terminator of x0D, the problem will persist.

My suggestion would be to follow Jim’s suggestion of getting the source system to provide record terminator of either CRLF or LF. Another idea might be asking the source system to send only one record per file, then you could use the EOF-terminator option in Cloverleaf.

The only other option I can think of today would be to write your own TCL script to open and read the file records as a protocol:upoc. Not a quick solution unless there’s already code out on Clovertech.

[Author]

Replies