.hl7 file type

[Christy Stepp]

I am trying to use fileset-ftp protocol in Cloverleaf to pick up a file named filename.hl7.  The file contains one HL7 ORU message with between each segment and at the end of the file.  I have tried multiple settings for the fileset-ftp properities and Cloverleaf will not read the file.

We are using UNIX, so I tried to look at the file in Cloverleaf.

When I type the more command with the filename I can see the message, it starts with MSH, and appears to be a complete HL7 message.  If I type the vi command with the filename the file appears as one @ per line with no other data.

Does anyone have any experience with the .hl7 file type?  The fileset-ftp protocol works for me in other fileset-ftp threads using .txt file types.

Topic

[Robert Kersemakers]

Hi Christy,

The way you describe the file, it should work with fileset-ftp. Just make sure you are reading it ‘single’ style.

Have you turned up the EO config to have some more logging on what is happening? And you say Cloverleaf will not read the file: is there an error given, or is the file just not picked up by ftp? The ‘Dir List Command’ is normally ‘nlst’, but if it’s something like ‘nlst *.txt’ it will only pick up *.txt files.

Your log file (especially with your EO config turned up) should indicate what is happening.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Russ Ross]

If the file is new-line terminated the setting fileset-ftp “nl” I would expect to worked.

However, once I was told my single message file was new-line terminated and it wasn’t, so I ended up using eof to get it to work.

I haven’t used the setting “single” which might have the same outcome.

If you do hcihd on the file in question you can see if it is new-line terminated, because you can’t tell in vi when there is only one message in the file.

You might also be able to use “wc” on the file and if number of lines is 0 then it isn’t new-line terminated.

Today I noticed for the first time a setting called “hl7” which I have no experience with, but wonder how that setting will consume a file, especially those HL7 files with one segment per line.

Russ Ross
RussRoss318@gmail.com

[Christy Stepp]

I was using style of hl7, and I also tried single.  The error in the thread is FTP operation failed: RETR response: 550.  I can see all of the files that the ftp parse tcl scanned in the directory, and I can see that it found 3 files that matched my regular expression.  I think error 550 is “cannot open file” (from google search!)  Not sure why Cloverleaf cannot open the files…

[Charlie Bursell]

Have you triedthe same thing from the command line?

[James Cobane]

Does the file have appropriate privileges for the ‘hci’ user to read/write from it?  Perhaps the user/process that is placing the file has different read/write privileges.

Jim Cobane

Henry Ford Health

[Christy Stepp]

I can manually FTP the file from the command line.  I have tried viewing the file with the vi command, it shows one @ per line, no other data.  If I save the file as a .txt file Cloverleaf will read it using the nl style setting.  I would like to know why Cloverleaf will not read it as a .hl7 file using the hl7 style or single style.

[Russ Ross]

I think I recall seeing one @ per line on files that have very long lines and can’t all be displayed in my vi view window.

Still would be curious to run hcihd on the original .hl7 file to know what is in it, since an editor like vi will not give you the complete picture.

When you say “saved the file as .txt” did you save the file using an editor like vi or “cp -p” from the command line or “mv” from the command line because they aren’t the same.

So when you do “diff” and “ls -l” of your .hl7 file and .txt file, are they are identical and the permissions are the same, too?

If so, that by itself is a head scratcher.

If you rename the .txt file that works back to .hl7 what happens?

I’m struggling to imagine the file extension would change the outcome of the fileset-ftp if the file contents, ownership, and permissions are identical.

Russ Ross
RussRoss318@gmail.com

[Christy Stepp]

I can see the content of the file if I use the hcihd command.  I saved the file as .txt by opening it in NotePad++ and using the save as text command. The diff command shows no difference in the two files.

[Robert Milfajt]

vi is showing @ because message is too long and vi has limitations.  I like to either use cat -v to show control characters or od -c (which is a file dump) showing characters, od -x shows hex values.

Once you can look at the dump of the file, you might have better data on why Cloverleaf is failing to read it.

Hope this helps,

Robert Milfajt
Northwestern Medicine
Chicago, IL

[Christy Stepp]

I was able to get Cloverleaf to read the files.  I have the style setting as single and the data type setting as Image/Binary.  Thanks to everyone for all of your help.

[Jim Kosloskey]

Christy,

Does this file have RTF or PDF imbedded in the messages?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Christy Stepp]

New problem.

The Fileset-FTP thread passes a single filename.hl7 file from the directory to the Xlate and it processes fine, as long as there is only one .hl7 file in the directory.

But if there are 2 files in the directory with .hl7 extensions I get an error:  550 filename.hl7 filename2.hl7: not a plain file.  I also get 550 filename.hl7 filename2.hl7: A file or directory in the path name does not exist.

I was expecting each message to process in the order they were in the directory.  The log file shows that the ftp_dir_parse tcl is passing back  new_msg{filename.hl7 filename2.hl7}  Then I see the errors from the ftp process.

Each file contains a single message.

Thread set as fileset-ftp

Style: single

Directory Parse: ftp_dir_parse (passing in parameters {OP FETCH} {TYPE REGEXP} {FILES .*.hl7} {DEBUG 9} )

Scheduling

Read Interval: 5

Scan Interval: 30

Max Messages: 1

[Russ Ross]

I wonder if setting messages to a higher number than one could be helpful and certainly not likely to hurt anything.

I keep mine to a rate of below 4,000 per minute so as to give cloverleaf 15 seconds of rest every minute on my platform setup and for slower integrations it might be as low as 200 per minute when I don’t want to overwhelm a slow downstream interface.

With such a frequent directory sweep, I wonder if the second file is still being written to when Cloverleaf attempts to retrieve it and might be causing the file not found error.

Make sure the .hl7 file(s) are ready for consumption and not being written to when the directory sweep occurs.

The simplest effect way to handle this I’ve run across so far is to have the file names whatever.hl7.pending while being written to and then have the source application rename them to whatever.hl7.ready or just whatever.hl7 to put Cloverleaf on notice the file creation has been completed.

Russ Ross
RussRoss318@gmail.com

[Jim Kosloskey]

Could it be that both file names are being presented as one:

550 filename.hl7 filename2.h7

Perhaps your dirparse routine is not returning a proper list.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Russ Ross]

Understanding what typically gets change when transferring in ASCII mode is helpful when determining is ASCII or binary FTP transfer mode is needed.

Different operating system use different line termination sequence and when transferring in ASCII mode those line terminators get convert.

So on Microsoft Windows or DOS a line terminator is , while on a UNIX platform the line terminator is .

Open a file up in an editor for viewing is okay but then saving the file can also super impose such undesired intelligence along with sending message through email and trying to use them.

hcihd is your fried or an editor that has a hex edit mode will give you the true story.

Since the hl7 segment termination character is a this can get in the way of a FTP ASCII transfer and might require a binary transfer.

Other times the source system might have one segment per line and some message formatting to combine back to one line is a common approach I use before sending them to xlate.

So when dealing with a new FTP integration I hcihd to see what I’m dealing with and that helps determine what approach I go with.

I could ask the vendor to make it right but typically take what I get and preprocess it before xlate if I get some variation but at least with hcihd I will be aware of what I’m starting with and knowing what I need it to be I just close the gap, which sometimes I accomplish with a binary transfer like you did.

I always do a manual FTP binary FTP the first time so I will be able to see with hcihd what I’m starting with.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Jim wrote:

Could it be that both file names are being presented as one:

550 filename.hl7 filename2.h7

Perhaps your dirparse routine is not returning a proper list.

So my question is are you using a dir parse routine or just letting Cloverleaf do the default built in dir parsing?

Russ Ross
RussRoss318@gmail.com

[Christy Stepp]

If TCL ftp_dir_parse returns one file it works; if it returns 2 files it does not work – it actually passes a blank message to the Xlate.  

When there is one .hl7 file in the directory

process log:  ftp_dir: test_ftp_oru: new_msg|filename.hl7|

msgDriverControl: {FILENAME /home/hci/temp/filename.hl7}

msg: MESSAGE IS HERE

When there are two .hl7 files in the directory

process log: ftp_dir: test_ftp_oru: new_msg|{filename.hl7 filename2.hl7}

msgDriverControl: {FILENAME /home/hci/temp/filename.hl7 filename2.hl7}

msg: ‘ ‘

I thought this was correct.  Shouldn’t it pass the first message in the list, process it, then pass the next message in the list and process it?

[Russ Ross]

Okay sounds like your dir parse routine needs a fix to properly handle multiple files.

Jim Kosloskey on our team has done all the dir parse work which we rarely need beyond the builtin cloverleaf dirparse so I’m not current on what dir parse needs to return.

Since Jim asked if your dir pase was returning one item with 2 files, I imagine dir parse needs to return a list of files names when multiple files exist.

I also recall the directory information returned to dirparse changed when we went from cloverleaf 5.6 to Cloverleaf 6.0 with expanded listing information that broke dir parse for us until Jim remediated.

I’m sure others can chime in on if my guess is correct that dir parse needs to return a list of filenames when multiple files need to be processed.

It is a common mistake to just concatenate things together separated by a space instead of using the list commands to create a list of items to return.

Russ Ross
RussRoss318@gmail.com

[Christy Stepp]

The TCL is returning new_msg|{filename.hl7 filename2.hl7} which I think is a list.

[Jim Kosloskey]

Have you echoed out whath the  list looks like you are receiving before you modify it?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies