Combining two messages into one

[Mike Strout]

I have an inbound results feed from our new lab system that is incapable of sending the textual result and its associated PDF in a single message, a requirement of my downstream systems. Instead, it sends an R01 containing the text and an R03 containing the encoded PDF. The sending system also can’t guarantee which message will arrive first.

My solution was pretty simple.

1. Which ever file arrives first, write it to a file with an extension of msg or pdf.

2. Which ever file arrives second, pull the contents of its mate out of the file, combine them into a single message and then write them to a different file location with a .hl7 extension

3. Delete the original message from the file system.

4. Finally, I have a inbound fileset local that polls for the .hl7 message, slurps it up and sends it on its way.

In the testing tool, this works perfectly, however, these two messages arrive almost simultaneously and the first file isn’t finished writing to the file system before the second goes looking for it, causing the second file to be written to the file system as well.

I have tried putting a sleep in the tcl just before it goes looking for the first file, but it seems that because the messages arrive milliseconds apart, the both sleep that 5 seconds concurrently and the same issue happens.

Any suggestions on how to delay the second message until the first is written?

Topic

[Robert Milfajt]

Does the system guarantee that if you get either R01 or R03 for a particular patient test, that the other follows right after??

Robert Milfajt
Northwestern Medicine
Chicago, IL

[Terry Kellum]

How about using a database with a “timed” proc to look for pairs.

Just write the messages to the database and let the timed proc pick out the pairs and put them together.  That makes the process async, and you can use a timestamp to make sure you are looking back no sooner than say 30 seconds.

If the duplicate is not found on “this” pass, it will be picked up on the next.

You could set a cycle time of say 2 minutes…

Once you build the message and save it in cloverleaf with a MID, delete the two source messages.

If you use sqlite, make sure that you vacuum on a regular basis.  Watch the file size to determine how often, unless there is an auto-vacuum…

[Mike Strout]

Robert,

Sadly, they can’t guarantee which comes first.

Terry,

I started with a Sqlite solution first, but I ran into problems trying to escape the sql reserved characters (like commas) in the textual result. While I love the new database connectivity in CL 6, sometimes it can be so frustrating to work with.

I ended up changing to a file strategy because it seemed clean and easy…until I ran into this timing issue.

[Terry Kellum]

Have you tried colon-variables?

Say you have a variable with quotes, commas and all sorts of nasty characters.

Let’s call this variable  $salad  .

You would then say:

db eval {INSERT INTO MyTable VALUES (:OldMid, NULL, :Timestamp, :salad)}

The Colon syntax causes sqlite to do the variable unpacking itself, preventing both weird character problems and preventing sql-injection issues.

[Russ Ross]

My simple solution I’ve been able to effectively find a way to work into my integrations one way or another to prevent consumption of files being written to is to name the file with something like whaterver.pending untit all writing to the file is complete and then rename the file to something like whatever.ready.

If I only look for the whatever.ready files then I never try and consume a whatever.pending file until it becomes ready.

Most of my file renaming is done with our batch model that are wrappers around the cloverleaf xlate.

Perhaps that is not your situation.

I did help another clovertechie that reached out to me by using the “Send OK Procs” TPS stack on the outbound thread to rename their file as whatever.ready and that worked for them, which might be an approach to consider for your situation.

From what I’ve seen from my little exposure to using the “Send OK Procs” TPS stack, my interpretation is that the most recent message has been delivered to the output thread (file in this case) and this is your chance to do something at that point in time, knowing the last message has been successfully written to your file in its entirety.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

I still envision a potential design consideration in what you described where neither of the 2 related messages see a visibly whatever.ready file because of timing issues.

Regardless of what approach is used this consideration might manifest itself and will need a solution for that case.

One thought I had for that is the whatever_R01.ready and whatever_R03.ready files can accumulate in a que directory that gets swept to process frequently the que directory for whatever.ready pairs that are both there.

For that to work the whatever part of the file name will need to be unique and match for the R01 and R03 messages received that belong to each other.

Russ Ross
RussRoss318@gmail.com

[Mike Strout]

Thanks all for your suggestions. I took a step back and decided to do the combining on the fileset connection that sends the composite message outbound.

1. txt or pdf message arrives from lab system

2. I use “msgmetaset $mh DRIVERCTL “{FILESET {{OBFILE $msgType/$accNum.$msgType}}}” to write the the message to a file in a txt or pdf folder.

3. I use a fileset local to pull each txt file from the directory.

4. If the associated pdf exists in the pdf directory, I open it, combine the txt and pdf, send the message on its way and then delete the pdf file.

5. If the associated pdf doesn’t exist in the pdf directory, I just write the txt file back into the directory it came from and set the disposition to KILL.

The only down side to this strategy is that the inbound SMAT only contains the txt version of the message, not combined version. This leaves me with no good way to see the entire combined message before it goes through the Xlates for the two OB routes. Not a deal-breaker though. All told, it is a pretty simple solution, way easier in my opinion than using an Sqlite DB.

[Jim Kosloskey]

If you want to see the combined message in a SMAT file, route it to intermediate thread(s) then on to the destination(s).

The SMAT for that/those thread(s) will show the combined message.

I would suggest you place the 2 ib MSH-10 into a combined field (may be too long for MSH-10) so that you can track the combined back to the originating messages by MSH-10.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies