Readin PDF files and sending imbedded PDF trough interface

[Karl Garen]

Hi,

Has anyone built an interface where a link to a PDF document imbedded into an HL7 message was received by Cloverleaf (or other related technology) and then convert to an imbedded PDF in the HL7 we could pass to a receiving system?

If so, what was your experience – good, bad, don’t even go there, pitfalls etc.

Thanks

Karl

Karl Garen
Sr. Programmer Analyst
University of Vermont Medical Center
Burlington, Vermont

Topic

[Karl Garen]

To clarify, the link would be a file link presumably accessible by the interface engine for reading and moving into an HL7 message.

Karl Garen
Sr. Programmer Analyst
University of Vermont Medical Center
Burlington, Vermont

[Daniel Lee]

I built one that is not in production yet.  It was actually much easier than I anticipated.  The technical process of picking up a PDF from a link and embedding it in Base64 is pretty easy in Cloverleaf.  There are some other things I’ll give you to keep in mind.

1)  I like to sometimes open the message in the SMAT file and edit the data in the HL7 messages to resend it.  Some of the HL7 messages got too big for me to edit in SMAT.  They also seemed too big to test using the Cloverleaf test tool (sometimes they worked sometimes they didn’t).  I never pursued why and just tested them by resending transactions and edited them at the OS level.

2)  It may be possible that the sending system is not done writing the PDF to the shared file location by the time Cloverleaf receives the messages and tries to embed it.  This wasn’t an issue for me but it could happen.

3)  A lot of receiving systems have a total message size limitation and a message size limitation for the OBX segment in which case you’d have to split the PDF up into multiple ORU messages and multiple OBX segments. Even if they can accept it, you’ll still have to make sure you change your HL7 variant to set the OBX-5 size to unlimited as those embedded PDF documents get pretty big.

That’s all I can think of off the top of my head.

[Steve Carter]

Karl,

We built this type of interface several years ago and it’s working just fine.  As Daniel already pointed out, it is relatively easy to implement in Cloverleaf, but can have some pitfalls.

1.  SMAT isn’t an issue for us as we don’t use it.

2.  If the PDF isn’t available, we just suppress the message and send an email alert.

3.  We have several options for delivery of results that contain one or more PDF’s.  We built a custom Z-segment that breaks the encoded data into 50K blocks.  We also can delivery it in the OBX or in a separate file if the receiving system doesn’t use TCP/IP.

4.  One of the things that we didn’t do (that I’d like to implement at some point) is to not embed the PDF until it is exiting the engine.  That will help performance as it will keep the overall message size down.  There really isn’t much reason to choke the engine with gobs of encoded data when it’s just passing through.  This is something that I would definitely recommend.

I’ll be glad to try to answer any other questions that you might come up with on this topic. 🙂

Steve

[Bob Richardson]

Greetings,

When you folks say “it is pretty easy to embed a PDF file” into a Cloverleaf message from a link, do you have something like the Data Integrator implemented for Databse lookups?

Right now we do pass Base64 encoded PDF (in an OBX) from applications

that send them to Cloverleaf.  Any PDF links received (in an OBX) are passed to another application (OnBase) that does the PDF report retrieval.

Just interested in the “how” here.

Thanks.

[Charlie Bursell]

I have done a few of these with little or no problem Karl

I found it easier to use TclCurl for the FTP because of the not ready issues already disussed.  However when not ready we placed in a sqlite database for later retry.

I find it *MUCH* easier in a IB or route proc than in Xlate.  But that is just me.  Others may differ

You know where I am

[David Barr]

We’ve got a few interfaces that receive HL7 with a link to a file on server1. Our Cloverleaf thread downloads the file from server1 and uploads it to server2, then we build a new HL7 message with a link to the file on server2 and send that message to an HL7 TCP/IP receiver running on server2. These interfaces are complicated, but they work very reliably.

[Steve Carter]

In my case, the PDF gets embedded into the HL7 outside of Cloverleaf via a tcl script.  At a high level, here’s how our PDF result flow works:

1.  Tcl script (outside of Cloverleaf) is used to determine which Cloverleaf site should receive the file (most processing is file-based).  It also identifies the need for PDF retrieval and places it into a separate directory.

2.  Tcl script (outside of Cloverleaf) reads the ‘link’ from the OBX and pulls the raw PDF from a separate server via FTP.  It then encodes the PDF and places it in the HL7.  The file is then moved to the appropriate site.

Hope this helps to clarify things.

Thanks.

Steve

[Daniel Lee]

I guess saying it was “easy” is relative right?  In our case the sending system is sending us the path to the PDF from the prespective of the sending system’s server (i.e. C:….).  Then we had the network guys mount a Windows share on our Cloverleaf AIX server.  From there I just picked up the file from that share and encoded it in the interface.  See, easy!

[Karl Garen]

Wow.  Thanks everyone.  Great stuff.  I like the NFS approach to get access to the files.  We have used NFS on some of our systems.

Karl Garen
Sr. Programmer Analyst
University of Vermont Medical Center
Burlington, Vermont

[Kamal Bahrani]

Hi Guys,

I’m facing an issue related to base64 encoding and would appreciate if anybody can help me out. I’m getting the pdf image encoded in binary format in ZPD segment and its encoded in uuencode. The receiving system needs it in base64 to display correctly and using tcl I’m converting from uuencode to base64. Tcl provides packages for this but when I encode it in base64, the resulting binary pdf is corrupt and not opening. I tried decoding using online decoders but not working. Please let me know if anybody knows abt it.

set pdf [base64::encode [uuencode::decode [parse_msg ZPD_3_3]]]

[Charlie Bursell]

Is the file corrupt if you simply uudecode it before the Base-64 encode?

[Robert Milfajt]

You might want to try it in steps.  First get the data out of the segment and into a file on UNIX and decode it.  See if you can open it.  If you can, base64 encode and decode it and see if you can run it.

Many things could be going on here and you need to validate the source first and work your way forward until you find where it breaks.

Good luck,

Robert Milfajt
Northwestern Medicine
Chicago, IL

[Anonymous]

Hello Kamal,

Kamal Bahrani wrote:

Hi Guys,

… and its encoded in uuencode. …

Before you perform a uudecode, check if there are already some chars additionally encoded.

The default result of a uuencode has after every 80 chars. As the >CR> are not valid chars in an HL7 message, they must be encoded again, before the file can be filled into an HL7 message.

Just check the thread “Muse vendor sending ORU with embedded pdf” for more info. (https://usspvlclovertch2.infor.com/viewtopic.php?)t=5658&highlight=uuencode+pdf

[Abe Rastkar]

We have an interface that receives PDF messages from surgical pathology Coded in MIME embedded in HL7. We send the messages to the EMR system where it is translated back to PDF.

[Mike Taylor]

Hello,

   I am a newcomer to Cloverleaf and Tcl and could use some assistance.  I am attempting to encode a PDF and send this through HL7 to the receiving system which will decode it.  I receive a message from Xcelera with the name of the PDF in the message.  We then get the PDF through FTP and place it in a directory so it can be opened in Tcl.

   I have already set the OBX-5 to unlimited so the size is taken care of.  In the Xlate I am passing the filename to the tclproc to open the file.  I can open the file however I can’t seem to get this to work past that point.  Once the file is opened I am trying to encode the PDF and pass this back to the Xlate.  I have reviewed the various posts on how to do this through Tcl without success.  I saw Steve is using a similar process.  Would someone have working code they would be willing to share?

   If needed I can provide the various iterations I have tried but was hoping a working example would be best in the interest of time.

Thank you,

Mike

[Daniel Lee]

Here’s all there is to my code.  It’s a call out proc that uses the path as an input paramater and the file name as $xlateInVals.  $xlateOutVals is the field where the encoded PDF will reside.

Code:

proc xltPROVencodeFile_v0 {path} {

   upvar xlateId       xlateId      
    xlateInList   xlateInList  
    xlateInTypes  xlateInTypes  
    xlateInVals   xlateInVals  
    xlateOutList  xlateOutList  
    xlateOutTypes xlateOutTypes  
    xlateOutVals  xlateOutVals

#    global HciSiteDir

   set file [lindex $xlateInVals 0] ;# Get the PDF file name from xlateInVals
   set fileName “$path/$file” ;# Set the whole path to the PDF file

   set fi [open $fileName r] ;# Open the PDF file for reading
   fconfigure $fi -translation {binary binary} ;# Set the file channel to a binary read
   set en64Var [encode -wrapchar “” [read $fi]] ;# Base64 encode the PDF file into $en64Var
   close $fi ;# close the PDF file

   set xlateOutVals [list $en64Var] ;# Write the encoded PDF to xlateOutVals
}

[Mike Taylor]

Daniel,

   Thank you for the code example it helped greatly.  I made a modification since I was pulling in the full path with filename from the xlate but other than that I ran the code as you provided.

   When I sent a message through I received the following error:

error message

Invalue first is: EmptyFillerOrderNumber_U2837_Stress{1}_09_05_12_0750a.pdf

pdfmap is: EmptyFillerOrderNumber_U2837_Stress{1}_09_05_12_0750a.pdf

encode value is: /quovadx/qdx5.7/integrator/cl_test_site/MikeT/EmptyFillerOrderNumber_U2837_Stress{1}_09_05_12_0750a.pdf

input: file300

[xlt :xlat:ERR /0:    ORU_xlate:09/11/2012 15:36:17] [0.0.33714269] Xlate ‘ORU_From_Xcelera_To_Cerner_MDM.xlt’ failed: Tcl callout error

erroCode: NONE

errorInfo:

wrong # args: should be “encode text”

   while executing

“encode -wrapchar “” [read $fi]”

   (procedure “PDF_Encode_Base64” line 18)

   invoked from within

“PDF_Encode_Base64”

 I then made another modification that is now working and doing the encoding.  Here is a copy of what is working for me now.

Code:

package require base64

proc PDF_Encode_Base64 {} {

   upvar xlateId       xlateId      
    xlateInList   xlateInList  
    xlateInTypes  xlateInTypes  
    xlateInVals   xlateInVals  
    xlateOutList  xlateOutList  
    xlateOutTypes xlateOutTypes  
    xlateOutVals  xlateOutVals

#    global HciSiteDir

   set file [lindex $xlateInVals 0]      ;# Get the PDF file name from xlateInVals

   set DEBUG 1

   set fi [open $file r]         ;# Open the PDF file for reading

   if $DEBUG {
      echo “input: $fi”
    }

   fconfigure $fi -translation {binary binary}   ;# Set the file channel to a binary read
   set en64Var [::base64::encode [read $fi]]   ;# Base64 encode the PDF file into $en64Var
   close $fi            ;# close the PDF file

   if $DEBUG {
      echo “output:  $en64Var”
    }

   set xlateOutVals [list $en64Var]      ;# Write the encoded PDF to xlateOutVals
}

[Mike Taylor]

I have successfully created the message with the Base64 encoding and tested the message by adding a decode to create a new PDF.  When doing a compare there are now differences.

  The problem I have is that the message does not leave Cloverleaf, it remains in a Pending state.  There are no errors so I can’t tell where the error is at.  I have smaller text messages going through the same thread so I know this piece is working.  Any ideas where to look as I don’t get any errors?

[Jim Kosloskey]

Mike,

Have you run the Xlate in the tester with the same message and the file and put the output of the Xlate tester to a file?

If that all seems to work correctly, I would look at the file with a hex dump (you can use hcihd which comes with Cloverleaf) to make sure everything looks good.

Then I would try resending the file just created in the Xlate Tester out the outbound thread and see what happens. If still no joy, I would then set up a file protocol thread and resend the message created via the tester to that thread.

In all cases, if you have not done so, try turning the EO up all the way and closely read the log – there may be some indications in there as to what is going on.

Are there any Tcl procs on the outbound thread or post Xlate? If so, I would start removing those to see if a difference is detected.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies