How to overcome OBX-5 size limit?

[Rainer Hurle]

From a medical application we send encapsulated PDF reports (Base64 encoded) via HL7 in OBX-5 field, as required by the IHE Cardiology Technical Framework. Those Base64 strings exceed the 64K limit of the OBX-5 field (IHE says: “An actor… must support encapsulated data with a length beyond the nominal 65536 byte limit of the OBX-5”). So, currently I’m having the problem that CloverLeaf truncates strings that are longer than 64K. In the HL7 Configurator I can set the length of OBX-5 manually to a maximum length of 100000 bytes, which does not really help, because this is still to small.

Is there any way to have CloverLeaf generate OBX-5 fields of unlimited length?

Rainer Hurle

Topic

[Jim Kosloskey]

Rainer,

You can change the field length in the OUTBOUND HL/7 variant definition (I expect you have a variant defined).

I cannot recall if a value of zero or a value of -1 makes the field indeterminant in length, but one of those values works. I had to do this once before.

Jim Kosloskey

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Rainer Hurle]

Jim,

I tried as you proposed. I created a variant and used the “HL7 Configurator” to modify field 00537 and segment OBX-5. In the “Field Length” edit control, I cannot enter a negative value. So, after setting field length to “0” and replacing the field in the OBX message, I resent the message: OBX-5 in the HL7 message is empty!

Am I doing something wrong? I use HL7 2.3.1 and Cloverleaf 5.2, if that matters.

Rainer

[Anonymous]

Rainer

OBX:5 has maximau 65536

Please see the HL2.3 spec here. Perhaps HL2.4 may have higer value.

Thanks

Reggie

B.1.55 OBX – Observation/Result

Seq# Item# Name OPT DT Len (Max) Reps Sender Variable Receiver Variable

1 00559 Set ID – OBX SI (10)

2 00676 Value Type C ID (2)

3 00560 Observation Identifier R CE (590)

4 00769 Observation Sub-Id C ST (20)

5 00561 Observation Results C ST (65536) ( )

6 00562 Units ST (60)

7 00563 Reference Range ST (10)

8 00564 Abnormal Flags ST (5) ( )

9 00639 Probability NM (5)

10 00565 Nature of Abnormal Test ID (2) ( )

11 00566 Observ. Result Status R ID (1)

12 00567 Date Last Normal Value TS (26)

13 00581 User Defined Access Checks ST (20)

14 00582 Date/Time of Observation TS (26)

15 00583 Producer

[Jim Kosloskey]

Rainier,

Try leaving the lenth empty.

I am on vacation and I can try this when I return next week – but I recall you can set the value to infinite (unless this has changed in a later release).

Jim Kosloskey

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Rainer Hurle]

Jim,

I’d appreciate your advice on how to set this value to “infinite”. I tried several methods but didn’t succeed (leaving the field blank causes Cloverleaf to use a default of “0”). Thank you.

Rainer Hurle

[Traci Zee]

Ranier,

I use the HL7 configurator to change the OBX-5 field to something other than a default and save it.  This creates an entry in the ‘fields’ file for that variant.

Then, I use an editor on the fields file to have a length more than 100000.  ‘Infinite’ is a relative term and I expect you don’t really want say a 100 MB pdf to go through the engine.  I’d be worried that something that large would cause bad things to happen.  I just give it a extraordinarily large value for the pdfs that I expect to get in my lab results.

You’ll need to do this with both the inbound and outbound variants.

Traci Zee

Emdeon Business Services

678.231.7201

[Rainer Hurle]

Traci,

your solution works for me!

I manually edited the “fields” file and set the length to ‘-1’ (as Jim proposed). Now, inbound and outbound size seem to not be limited any longer. I tested with a pdf file larger than 100000 characters which is now transmitted correctly.

Many thanks to you and all who replied.

Rainer Hurle

[Jim Kosloskey]

Rainer,

Aha!!!

So there is a bug in the Platform 5 GUI!!

I recall when I had this working I was on 3.3.1.

Please report this to Cloverleaf(R) support to be included as a bug. The GUI should allow the specification of -1 as that means ‘no limit’.

Quovadx- if you are watching- we should not have to use an editor to accomplish what used to be supported by the GUI (prior to Platform 5).

Jim Kosloskey

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Gary Atkinson]

So, is this a bug in 5.5?  Changing the field len in the field file to -1 shows as zero in the gui??

[Jim Kosloskey]

Gary,

The GUI (5.6) still does not let you put -1 in the field length for HL/7 variant so I suspect 5.5 won’t either.

You will need to use a text editor to edit the field in the fields file in the appropriate formats directory. In order to get an entry you can c hange, you might go into the gui and change the OBX-5 filed length to something other than its current value (but not -1). Then go into the fields fiel with the editor and change the length to -1.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Gary Atkinson]

Still shows as 0 in GUI when LEN -1 in the fields file.

[Jim Kosloskey]

Gary,

See my reply to your other post on this topic.

I still think this is a bug in the GUI but it will work.

Please contact support and ask to report a bug.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Troy Morton]

Hi Rainer,

Another option is to send only a pointer to the PDF file in HL7 through the engine, then on the backend (OB TPS) write a proc that puts the PDF back into the message just before its sent to the desitnation.

Sending large messages through the engine can be problematic to performance and the amount of memory or swap space used by Cloverleaf especially in a high message volume environment.

Troy

[Rob Abbott]

I have submitted this as a defect.

I tested on several 5.x versions and none currently allow a minus sign to be entered in the field length text box.

Current workaround is to edit the fields file manually if you really want -1, or enter a very large number in the GUI.

Rob Abbott
Cloverleaf Emeritus

[Kevin Crist]

I have a question about this, we are in the process sending a pdf’s through the engine, this is the first time we have done this and there are a few more systems to do after this. One vendor says the size of their pdf can be 1meg…to me this seems a little big but i am new on this venture, will this cause some system issues or will cloverleaf even take it? Just curious about how to go about these adventures. We are running CL5.4.1 on  a HP 9000 rp4440 Server.

2 – Dual Core Pair Processors (PA8900)

12.288Gb RAM

6

[Troy Morton]

Hi Kevin,

In my opinion, the most significant factor in Cloverleaf performance when you’re workign with large messages is the Recovery Database.

[Sam Craig]

Rob Abbott wrote:

I have submitted this as a defect.

I tested on several 5.x versions and none currently allow a minus sign to be entered in the field length text box.

Current workaround is to edit the fields file manually if you really want -1, or enter a very large number in the GUI.

One other thing Rob, the GUI will only allow a number up to 100,000.

[Jim Kosloskey]

Rob,

Thanks for submitting this as a defect.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Kevin Crist]

Does anyone have any examples or some details about how to setup a pdf with a pointer. i am wondering if this will be the best way to go but have not a clue how to get there.

Thanks.

[Rainer Hurle]

Hi Troy,

thanks for your suggestion, right now the trick with the manual adjustment of the length setting works fine for us.

Troy Morton wrote:

Hi Rainer,

Another option is to send only a pointer to the PDF file in HL7 through the engine, then on the backend (OB TPS) write a proc that puts the PDF back into the message just before its sent to the desitnation.

Sending large messages through the engine can be problematic to performance and the amount of memory or swap space used by Cloverleaf especially in a high message volume environment.

Troy

[Gary Atkinson]

Kevin I think it depends on the receiving system.  For example, Cerner requires I specific set-up in OBX:5.

[Richard Hart]

Hi Kevin.

We are using the Cloverleaf web services to pass documents (using MTOM),  we strip out the document when we receive the message and grab it again when we send the message out.

In HL7, the document could be stripped out of the HL7 field and the field replaced with a unique ID by the receiving application.  If this is completed in a TCL module, then the message is a string and there are no field size issues.

When the message is sent out, the last action (pre-write?) could grab it and put it back again.

You may need to add some housekeeping for the file clean-up.

read_file and write_file are useful for this and the uuid module can provide unique ids.

[Russ Ross]

Kevin Crist wrote:

Code:

Does anyone have any examples or some details about how to setup a pdf with a pointer. i am wondering if this will be the best way to go but have not a clue how to get there.

The time I did this sort of thing, I was fortunate enough to be included in the design early on but after everyone had gotten hostile because they couldn’t figure out how to effectively send across massive amounts of giant documents.

I was able to calm down the troubled project team by convincing them to use the pointer method.

I still remember how crazy and upset they had become with frustration prior to my participation.

That first brainstorming session was quite intertaining being the only sane and calm person in the room, while all the others had snapped with frustration and verbally imploding as a group against each other.

I even think some restraining orders were issued.

I was able to get both parties to agree on using a document pointer and not have the source system application send the document in the message from the start.

So happily the source application was made to send a pointer to the document, which was either a URL or a fully qualified filename to a shared drive including the path.  

I’m thinking it was a URL but this was too many years back for me to remember specifically.

I was aware of the -1 for unlimited field size back then but we all agreed that would take more than its fair share of the cloverleaf integration engine, plus we knew a document could have HL7 delimiter characters in it and I didn’t know about base64 encoding way back then as a way around this problem.

The downstream receiving application was modified to use the document pointer to retreive/access the document as needed.

Unfortuately, I’m not always so fortuante to be involved early on in the integration design and have to deal with legacy applications.

Consequently, I do have a DMS to TRAN integration that does send the document in the message.

We discovered that some people are nuts and create documents so ridicuously large that infinite size is almost too small for them.

So I gree that you still want to limit the size to whatever the largest limit is agreed on because someone will find a way to exceed it.

My approach was to check the documents size and set a flag in the user meta data to note the doucment had been truncated because it was deemed too large.

Then the desired appropriate actions could be taken when encoutering the user metadat flag being turned on.

In my case wording was added to the end of the document stating it was too large and had been truncated and email/page alerts would be sent out to support personnel to resolve.

Here is a URL I posted illustrating exactly how I set the user metat data in the xlate and how I retrieved it downstream in the outobund protocol UPOC:

<a href="https://usspvlclovertch2.infor.com/viewtopic.php?t=1367&&#8221; class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?t=1367&amp;

Russ Ross
RussRoss318@gmail.com

[Charlie Bursell]

Kevin Crist wrote:

Does anyone have any examples or some details about how to setup a pdf with a pointer. i am wondering if this will be the best way to go but have not a clue how to get there.

Thanks.

Take a look at MDM message type.  The TX.16 field is defined like:

{ITEM 00927} {NAME {Unique Document File Name}} {LEN 30} {TYPE ST}

[Jim Kosloskey]

Kevin,

Since you are concerned about using the Reference Pointer in the OBX segment, here is some information:

The use of a Reference Pointer in the OBX requires that OBX-2 field be valued with ‘RP’ (RP is the Reference Pointer HL/7 Data Type).  The OBX-2 field indicates the Data Type of the OBX-5 field.

Now the actuual Pointer will be in the OBX-5 field.

The RP Data Type has a specific structure. That is covered in detail in Chapter 2 of the HL/7 standard (I think it is Chapter 2 of any 2.x version).

I am pretty sure the RP Data Type is not specified in version 2.1. I know it exists in version 2.3 and beyond.

Essentially this is the construct of the RP Data Type:

Component-1 : Pointer (ST Data Type Component). This is where the actual pointer goes.

Component-2: Application ID (HD Data Type Component)

   This consists of 3 sub-components (typically separated by the &):

           Namespace ID (IS Data Type)

           Universal ID (ST Data Type)

           Universal ID Type (ID Data Type)

   This component is used to identify the Application that houses the object pointed to by the pointer.

Component-3: Type of Data (ID Data Type). There is a Table in the standard that indicates possible/recommended values but this essentially states the general category of the data that will be foind in the object pointed to by Component-1.

Component-4: Subtype (ID Data Type). There is another Table in the standard that contains possible/recommended values. The standard also describes how to specify local entries to both the Type of Data and subtype Tables for exceptoins to what is in the respective Tables. This is a component which further clarifies Componenet-3. For example if the code in Component-3 indicated Image Data Component-4 might indicate RTF (Rich Text Format).

If you decide to use a Reference Pointer in the OBX – might as well do it corectly and follow the standard.

You have 2 options as I see it within Cloverleaf(R) Xlate:

1. Change the Data  Type for field ‘Observation Value’ in the inbound and outbound variant from ‘ST’ to ‘RP’ and then you will see all of the components that make up the RP Data Type (but not the sub-components because the Xlate does not present those in the message tree).

2. Leave the ‘Observation Value’ Data Type alone (ST) and use the component and subcomponent notation by typing it in yourself with each reference.

There may be other resolutions…

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Kevin Crist]

Has anyone ever worked with BDI software. We are sending pdfs to our Mckesson HPF system and they have installed BDI on that server and apparently it reads the pdf’s and puts them where they need to be. Has anyone worked with this?

[Peter Heggie]

We are also interested in converting a BDI file to HL7 in order to get a transcribed report out of the file. Does anyone have experience converting a BDI file?

Peter Heggie

[Matthew Rasmussen]

Hello, (GUI 5.6.5P) I have tried the -1 approach as well we the 100000 approach, but my OBX-5 is still getting capped @ 64K.

[Jim Kosloskey]

-1 should work on 5.6 using an editor not the GUI HL/7 Configrator- just make sure you don’t touch it in the GUI configurator

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Kumar Ananth]

We have a tcl proc that checks the message and throws it out if the size is larger than 64K. I had to remove that as I had to pass HL7 BASE64 PDF from a vendor to Cerner. I’ve created a new variant and bumped the OBX-5 to handle the message size in CL. Looks like Cerner is not able to digest all of OBX-5 content which is over 64K and vendor can’t break it into repeating OBX-5. So, I need to send the OBX-5 content as repeating OBX-5 with each OBX-5 less than 64K. Any suggestions on how/where (xlate or tcl) to accomplish this? Anyone willing to share your script or provide a starter script?

TIA

[Author]

Replies