File pickup vs manual resend

[Ron Swain]

Recently we had to load a bunch of records from a flat CSV file.
We have an external shell script that converted the file into HL7.
We tried 3 different ways to then load the message thru Cloverleaf into our EPIC system.
1) Each line in the flat file was placed in a single .msg file in a directory to be picked up by Cloverleaf.
2) Created a single file full of HL7 messages in a directory to be picked up by Cloverleaf.
3) Opened the Control setting for the inbound interface and used the “Resend” function to send the file.

We were very surprised that the last method was the fastest (on a factor of about 8 times faster).  Is this expected or should either of the 1st 2 methods ben faster?

Topic

[Jim Kosloskey]

Methods 1 and 2 performance is controlled by the ‘Read Interval’ and ‘Max Messages’ settings of the fileset-local protocol (I am assuming Fileset-local here).

If you ran with the defaults, the performance would be slow. I am not sure what goes on under the covers for method 3 but I am pretty sure you can improve the performance of the other methods with appropriate settings.

Did you try various settings?

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 60 years IT – old fart.

[Ron Swain]

Jim Kosloskey wrote:

Did you try various settings?

No, but will try some experimentation this week.  Thanks.

[Jason Russell]

I agree with Jim. Your scan interval controls how often (in seconds) it looks, read controls how often it reads a new set of lines in the file, and number of messages controls how many messages it reads at a time. I usually keep ours at 1 scan, 1 read, and 10000 messages, but we rarely load large datasets, and when we did (millions of messages), we had to adjust so we wouldn’t cause data delays due to cloverleaf processing inbounds first.

Beyond that, there is a bit of a delay even picking up the files, usually a few seconds (it also depends on how large the file is).

with the resend function, you’re injecting the messages directly into the engine skipping all of the file reading delays.

But with the defaults, method 1 would take forever (picking up a message, waiting 30 seconds then picking up a new message). Method 2 wouldn’t be much better (picking up a message, reading a line then waiting 5 seconds to read the next message).

If you’re looking to pick up a single file that you control when it picks up, file is another option that should be faster. You would control when it picks up by restarting the thread. Once it reads the file it’s done until another file with the same name is placed and the interface restarts.

[Ron Swain]

Thanks for the feedback. Adjusting the setting made a substantial difference.

[Jim Kosloskey]

Thanks for sharing your results – this will be helpful to others.

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 60 years IT – old fart.

[Author]

Replies