Reading in large files

[Gene Millard]

On Cloverleaf we recently read in a file that had about 55000 HL7 messages.

There was a BHS segment at the beginning and a BTS segment at the end of the file.

It took 21 hours to process.  It took awhile for the file to be read in.

We are wondering if there is something we can do to speed up the process and if there is any way we can see that the file is still being read.

I am wondering if we have the Fileset Local Inbound Pane setup properly.

Our setting are as follows.

Style :  eof

CRNL Convert : None

Order : Timestamp

TPS is checked

Read Interval : 5

Scan Interval : 30

Max Messages : 5

Output is pdl-tcpip to our EHR.

We will be reading in files of this size or larger in the future.

The input file is a windows file.

We are on Unix.

The Guthrie Clinic
Sayre, PA

Topic

[Jim Kosloskey]

You might try adjusting your Read Interval (which, expressed in seconds, determines how often a read of the file will take place – in your case a read every 5 seconds) and Max Messages (which indicates how many messages will be read at each Read Interval).

There is no set of ‘magic’ values but rather there will be a setting for your environment and situation which is optimal.

I think you definitely want to read more messagtes each Read Interval.

As it is you are having Cloverleaf read 5 messages every 5 seconds so really you are getting about 1 message per second so it will take about 55,000 seconds minimum to read the file. With 3600 seconds per hour 55,000 / 3600 = 15.7 hours minimum.

You probably want to get 3 messages or more per second (6 messages per second even better). 3 Messages per second might get you around 6 hours and 6 messages per second could get you to around 3 hours.

You want to also cosider the setting of the Read Interval because you want a ‘breath’ to be taken between read intervals. In other words you don’t want a queue to build up in Cloverleaf where it is not done reading the file in that read interval before the read interval expires and that depends on the power of your environment – which includes the I/O rate your system can provide as well as the CPU/memory combo.

Basically in your test environment use a smaller file and experiment with the number settings until you achieve a ‘happy’ result.

I hope none of my calculations above are very incorrect but the process I think is representative.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Gene Millard]

Thank you for the information.  I will adjust the max messages setting.

I do not think it matters about the Read Interval setting because the file does not start processing until the entire file is read in.  

Which brings me to the second part of my question.  While it is reading this file for several hours it looks like the thread is not doing anything.  Is there any way to show that the thread is still reading the file?

The Guthrie Clinic
Sayre, PA

[Michael Hertel]

Style :

[Jim Kosloskey]

Gene and Michael,

Sorry I blew right past the Style.

As far as knowing if the file is till being read, unless something has been changed in Cloverleaf 6.1 or 6.2 of which I am unaware there is no direct method I know of to know that.

However, if your values for the Read Interval and Max Messages are set correctly you should be able to observe the Route Arrow change colors (and the stats from the display change), and the destination thread Pending count rise and fall.

As long as that is happening, the file is being read.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

Regarding file read progress…

You could have a proc at the DirParse UPoC to email and/or record the situation that the file is about to be read (assuming only one file in the directory at the time of the scan).

Then at the File Delete UPoC you could have a notification the file is done reading.

That would give you the potential start and stop times. Until the delete happens the file ostensibly is being read but the monitoring of the integration as mentioned in an earlier post could be used to verify things are moving along.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Gene Millard]

I have set the style to nl and changed the tcl that was changing the text into HL7 messages.  I have also increased the number of messages read per scan.  The messages are now being processed while the file is being read and both threads show that activity is happening.  This has cut down the amount of time needed to process the file.

Thank you all for your advice.

Gene

The Guthrie Clinic
Sayre, PA

[Author]

Replies