Embedded PDF via Base64 failing

[Scott Caldwell]

My issue is I use a 3rd party tool to FTP down the file into a folder, Cloverleaf grabs the file but its not processing.  The file just disappears.  Nothing in the error log, and nothing other than notes that a file was found, opened & found EOF regarding the actual file being processed.  If  remove or otherwise truncate the OBX5.4 field the file processes just fine.  I verified the base64 encoding does provide a valid PDF file.  It appears to go thru the test tool just fine.

Attached are the log, the two files in question (one works the other fails), and the stdout from the test tool.

This is just a pass-through interface at this point, no TCL processing going on.  Its just picking up the file, and sending it through to its destination.

Any help would be appreciated.  Thanks.

Attachments:

You must be logged in to view attached files.

Topic

[Matthew Rasmussen]

Hi Scott,

Just to clarify, you’re wanting to read in a PDF file, then encode it into an HL7 message, right? If so, how many threads are you using there?  Give us a rundown of how you’ve got your threads set up, and what POC you’re using your decode proc.  Its possible the file is getting read in, there’s nothing to do.  So there are no errors, but nothing is getting done.

Also,  you mentioned, “If  remove or otherwise truncate the OBX5.4 field the file processes just fine.” – so what is the outcome when everything goes fine?

I have seen something like this when the PDF file size exceeds 65535 bytes, and there is a workaround, but that pertains to code that references an HL7 variant.  Is your code referencing a variant?

[Scott Caldwell]

Responses to other questions:

If so, how many threads are you using there? Just two, IB & OB, both fileset-local.

Give us a rundown of how you’ve got your threads set up:
IB: fileset-local, encoding:ASCII, Log: enable_all,TrxID Det.: HL7, HICIStatic route(Raw) with Procs: intrinsiqtest which i was using to throw echos as needed, and seq_FileOut.

OB: fileset-local, encoding:ASCII, Log: enable_all,Outbound only

and what POC you’re using your decode proc. Not sure what this means.

Its possible the file is getting read in, there’s nothing to do. So there are no errors, but nothing is getting done.  This works with all other messages & message types I have tried.  Its only failing with the one with the full embedded PDF in it.

Also, you mentioned, “If remove or otherwise truncate the OBX5.4 field the file processes just fine.” – so what is the outcome when everything goes fine?  It runs fine, file is written to the designated outbound folder.

Is your code referencing a variant? i dont think so?

• This reply was modified 4 years, 9 months ago by Scott Caldwell.

[Jim Kosloskey]

Do you have a log where one message was used with the base64 test did not create a file?

 

 

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

Log with eo turned up so we can see the message and all that is happening inside the engine I meant.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Scott Caldwell]

I have enable_all turned on.

In the log the edited file is named 1584545623256alt.dat – while the full one with the embedded PDF is named 1584545623256.dat

This never happens in the full file:

DRIVERCTL is set to: {FILENAME {../../../data/LocalSrc/ib_test/1584545623256alt.dat}}

Attachments:

You must be logged in to view attached files.

[Jim Kosloskey]

To me it looks like the file is found but EOF is located no actual data. Looks like a short read. If this happens when you control the placement of the file (you actually put it there not the foreign process):

What style do you have for the IB File and what is really the end of message terminator in the actual file?

Using a hex tool to look at the file in hex, assure the end of message terminator is not in the message.

If the style is EOF, make sure (again looking in hex) make sure that is not imbedded in the message.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Charlie Bursell]

How are you putting the file in the folder?  It may be that CL is reading the file before FTP is complete.  You could FTP the file to a temp directory then move to CL directory or perhaps get FTP via Cloverleaf.

I set up a test site with 2 threads.  I read in the full message via Fileset/Local and raw routed to an OB thread and then to a file.  Worked great!.  Did I miss something?

[Scott Caldwell]

The PDF is already encoded in the message.  That attachment is the HL7_PDF_full.txt file that I attached.  I have tried both FTP directly into the source folder, and copying it locally from another folder, both with the same results.

I do have enable_all in both the source and destination threads – and the resuling log was attached.  You can see in the log it finds the file, but nothing ever comes of it.

In the log the edited file is named 1584545623256alt.dat – while the full one with the embedded PDF is named 1584545623256.dat

This never happens in the full file:

DRIVERCTL is set to: {FILENAME {../../../data/LocalSrc/ib_test/1584545623256alt.dat}}

[Matthew Rasmussen]

1. What does seq_FileOut do?
2. So you are not doing any encoding, just moving the file from one place to another?
3. To Charlie’s point, try stopping the threads, dropping the large file in place, then starting the threads.  ie, make sure the file is fully formed before CL takes a swing at it.

[Scott Caldwell]

seq_FileOut writes the file to a folder.  All this time I thought it was native to the TCL library…apparently not?   Its attached.  Is it my culprit?

Attachments:

You must be logged in to view attached files.

[Jim Kosloskey]

That proc just changes the name of the OB File it does not actually do any File I/O.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Scott Caldwell]

OK so I removed all scripts and just run a file though, with the process stopped when  moved the file into the source folder.  I tried the ‘good’ one first, and it went just fine, and created a file named default in the destination folder.  Expected that result.

Then I put the real file in, it was picked up, but nothing happened.

Then I put a 2nd good file in the source folder, it was picked up and the default file was overwritten with that file.

So…back to square one for me.  LOL

Attached is the log from this iteration of attempts.

Another thing that has bothered me is the look of the test tool’s results.  Under normal circumstances, I only ever see the “tcl :out: INFO[…]” lines at the begining of the message, and the note of where it is being routed to.  As you can see in the attached screen shot, thats not the case with the unedited file.  Maybe a clue?

• This reply was modified 4 years, 9 months ago by Scott Caldwell.

Attachments:

You must be logged in to view attached files.

[Jim Kosloskey]

No attachment.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Scott Caldwell]

Here you go.

Attachments:

You must be logged in to view attached files.

[Jim Kosloskey]

So if you:

Stop the entire process.

Drop the IB file in the Source folder manually.

Then start the IB thread, you still get EOF found?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Scott Caldwell]

Correct.  Log of that sequence of events attached.

Attachments:

You must be logged in to view attached files.

[Jim Kosloskey]

What is the Style configured in CL for the IB File?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Scott Caldwell]

Not sure what the acronym CL stands for?

But the Style in the Fileset/FTP Local (TPS) Protocol Properties, in the Inbound area, is hl7.  That what you were asking for?

[Jim Kosloskey]

CL is shorthand for Cloverleaf. Yes that is the value I am talking about.

Are these files multiple messages per file IB or one per file?

I believe that style says the messages will be terminated with CR,NL (or LF). What could be happening is the message is not terminated that way. Using a hex tool to view the file (hcihd command line command will work), look at the ending 2 characters to see if they are CR,LF.

 

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Scott Caldwell]

That appears to be the issue.  I used an editor to append the 0D at the end of the file and it went through.  I have asked the vendor to make sure every segment ends that way (the rest of them do).

I’ll keep you posted.  Thank you all for your help.

[Jim Kosloskey]

<p style=”text-align: left;”>Each segment should be terminated with CR. Only the last segment of a message needs the NL as well as the CR.</p>
I am glad you have it working and the vendor will cooperate.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Charlie Bursell]

If you are just reading the file in and not doing anything with it why simply use single or eof?  Just treat it as a blob

[Scott Caldwell]

Can you elaborate?

I am still having issues getting it formatted so that it will cross automatically.

[Jim Kosloskey]

What Charlie is indicating is to use the EOF Style on the IB. Then the entire file will  be treated as a single potentially very large message. If you are not needing to break into the message to alter anything or check anything then that entire ‘blob’ will be moved through Cloverleaf as a single message to be finally written to the outbound file.

Charlie can clarify if I have that wrong.

If, on the other hand, you need to treat each IB message separately then each HL/7 segment should be terminated by hex 0D with the last segment of each message terminated with hex 0D0A.

That should handle things for you.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Scott Caldwell]

The EOF style will do the trick.  Turns out the vendor was hesitant to modify the file they were creating.  This is the first ‘public’ file-based interface I have had to create.  I would’ve just tried out FTP, but it had to be FTPs and I dont have the plug in for that.

I appreciate everyone’s feedback and support.  Thanks. Scott

[Author]

Replies