sub component separator fails Xlate when changed by sender

My two cents:

It’s really not changing it to a curly bracket, it’s instead returning a list.

{$KEY $VALUE}

It’s picking up the first character which is the curly bracket.

What is in the sub-component separator position of MSH-2 in the message from the sending system?

I misunderstood.

I did not realize the curly bracket was being done by the source.

I read the post as you received the ampersand and the Xlate was returning the curley bracket.

My bad. Nevermind.

😳

So if you populate the OB MSH-2 with the { for the sub-component separator it works?

So what ‘should’ happen IMO is Cloverleaf should change the incoming { to an & on output (since HL/7 specifies that component of that field having subcomponents) unless you change the OB MSH-2 inside the Xlate.

Now I know post 5.8.5 Cloverleaf was modified a number of times to more properly handle sub-components. During various versions of those changes there was no need to specify sub-component notation during a COPY (as I recall that kept changing and I do not know what was finally settled on). PATHCOPY on the field seemed to work most of the time in most of the releases. Sub-component notation always worked and appears to still work.

Also I think your past success probably stemmed from the fact that the variants default to the standard for the various separators.

What exactly is the ‘failure’ you are experiencing?

Is there a Tcl proc being used in conjunction with the COPY or is the QDX_Tcl_SplitList proc not one your shop invokes?

Personally if I know there are sub-components for a component I use subcomponent notation or a PATHCOPY for the component (hopefully that still works in 6.2).

I go that way basically because in CL 3.1 through 5.6 subcomponents were kind of not handled uniformly by CL but the COPY with sub-component notation did work and once PATHCOPY was introduced that would also work.

However, as long as Infor is trying to make the sub-component handling better it is my position when copying a field with components which is defined in the standard as containing sub-components, the sub-component separators should be handled appropriately without requiring sub-component notation at the field’s component level.

My questions regarding setting of MSH-2 is my attempt to find out if the Xlate was sensitive if the ob sub-component separator is set within the Xlate in 6.2 perhaps as a method to assist.

You are probably correct that the failing Tcl proc belongs to Infor.

IMO this should not happen. If you are so inclined, you could report this as a bug.  

I would but I don’t have Cloverleaf or Xtreme access right now.

as Jim says this looks like a bug, please open a case in Xtreme so we can get it addressed.  thanks!

Hello – just checking to see if anyone has an update as to when INFOR may put out a patch for this bug.  thanks

Working with Peter offline, I was able to reproduce the issue and Michael Hertel was dead on to the answer; although he may not have known it.

In the Copy stmt you are sending in two dates separated by a curly brace and then treating the entire component as a scalar value. In trying to extract the second date with the crange fn based on the assumption of the values length, you create the error. What is being missed in this case is the { is a reserved character in TCL. Cloverleaf knows this and escapes it for you with a backslash; Cloverleaf has many hidden jewels like this. This makes the subcomp separator two chars long in the xlateInVals data set.

When you run the crange function, your target text is actually “20160520{20171019” and the return text becomes “{2017101”, not what you have expected. This is in turn is passed into xlateOutVals, which makes it a malformed TCL list, which throws the XPM error.

You can simulate the error with this code in any Xlate COPY stmt:

Hi all,

Peter did open a ticket for this, but I also did find a work around to replace the ampersands with the word ‘AND’ and then replaced all the curlybraces with the ampersands.  

Initial testing looks good, I am going to do a compare and volume test before implementing, but this will put the message back to the way we expect it to normally be.  In example below, patient access team accidentally used the ampersand in the patient reason for visit field.

Before:

MSH|^~{|SOARF|J0A8|||201711011601||ADT^A08|

PV2|||^COLD SYMPTONS & EAR PAIN|||

GT1|1|….|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520{20170716||

IN1|1|194^POMCO|…|HEALTH|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520{20170716|SELF

Code used to revert the delimiter back to an ampersand:

       set curlybrace “{“

       set addlsubFieldSep [cindex $msg 7]       ;  #MSH2 input field

        if {“$addlsubFieldSep” eq “$curlybrace” } {

             regsub -all {[&]} $msg {AND} msg

             regsub -all {[{]} $msg {&} msg

Giving output as below:

MSH|^~&|SOARF|J0A8|

PV2|||^COLD SYMPTONS AND EAR PAIN||||~~~~FALSE|

GT1|1|…|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520&20170716||…

IN1|1|194^POMCO|….|||HEALTH|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520&20170716|SELF|…

We are not actually using or passing the second date field downstream, we are merely checking for the presence of one to see if the data has a termination date.

Any questions, let me know  8)

Thanks.. Brenda

This is something I have seen over and over since the 70’s when HL/7 first came into being.

The original authors of the HL/7 standard wanted to be able to allow the use of the separator characters within text and not be treated as separators.

How they decided to do that was to specify an escape sequence for the various MSH-2 values. That is what the escape separator is for in the MSH-2.

The original framers of the HL/7 standard (I recollect this actually existing in pre 2.x) understood that any parser would have issues with spuriously placed separators.

This is all very clearly specified in the HL/7 specification. For most of the versions of 2.x it was in the same section of Chapter 2 (2.7 to be specific).

In later releases that was moved and a new separator was added but it is still there.

IMHO the ONLY way to handle this properly is for the sending system to apply the proper escaping when building the messages. ALL other ‘tricks’ deployed simply are waiting for failure. In the 40+ years I have worked with Integrating HL/7 I have NEVER seen a fool proof non-standard resolution.

It is also incumbent upon receiving systems to un-escape properly. Although that can be accomplished easily and consistently by Cloverleaf but then the improperly placed separators become a problem for the receiving system’s parser.

Specifically, for this workaround as an example, what happens if a user of the sending system inadvertently uses { when they wanted ( or [?

The work around will change the spurious { to a & and now that will appear to be a subcomponent where potentially non should exist.

There is no way other than deploying the standard to accommodate all the possible permutations.

I submit if you never want to be worried about spurious separators causing you issues then pressure needs to be brought on the sending system to follow the standard – a least this rather detailed and specific part of the standard.

It can be done. The last project I worked on at MDACC was an upgrade to the Lab system. I forced the vendor to follow this portion of the standard and we never had an issue after. I got pretty beat up and certainly did not make a friend in the vendor but I would do it again.There is no unmarked land mine laying there for the Integration engineers that followed me.

I apologize for wandering a bit off of the specific issue – but this is related and I think important.

Anyway that is my .02 (may even be worth less than that to many).

Thanks Jim for your opinion and words of advice, I greatly appreciate it!  

In my example, I am only changing the { to an & when one resides in the MSH2 as the  sub sub field delimiter instead of the place where the ampersand is – this would indicate that it is not used in any other field in the message.  Now in the case if they used non standard delimiters and they used the ampersand as a different qualifier in MSH2, then we would be in trouble – I should add additional code to make sure the other three delimiters are NOT an ampersand to ensure that doesn’t happen.  When dealing with humans (the patient access team), there is always a chance of error in freeform text fields and we cannot go to the vendor in that case – only re-iterate the importance of not using special characters when this issue arises.  In the other instance where this happened we did ask that vendor to NOT send the ampersands in the data/message for our messages.

We will continue to monitor this, thanks everyone for your inputs!

Brenda

Brenda,

The HL7 message is not malformed. The sending system is using the curly brace as the sub-comp separator, which does follow the HL7 guidelines. Cloverleaf is supporting the curly brace as a sub-component separator as well. The only error I found here is the extraction of an HL7 component, the assumed length of the component field, the assumed presence of a single character sub-comp separator and the use of a TCL crange function leading to a malformed TCL list passed back via xlateOutVals.

The xlate solution to  this issue was already mentioned by Peter in a previous post. It will fix the issue and work 100% of the time. Bug free.

Peter:

But the other method which works, is if I leave the { in MSH2 as the sub component separator, and then modify the COPY statement to specify the subcomponent like this

example a: 6(%g1).IN1(0).#16(%f1).[9].[0]

instead of our original code:

example b: 6(%g1).IN1(0).#16(%f1).[9]

then the translate works also.

This leads me to believe that Cloverleaf is working correctly (when the COPY statement specifies the sub component as in example a) and we have been depending on ‘beneficial’ behavior which actually should not be relied upon if there are sub components with certain delimiters.

In effect, use Cloverleaf’s COPY command with the sub-comp path to extact that second date value and all works as expected and desired, regardless of the sub-comp character used by the sending system.

I did not see any mention of other issues with the message and it’s curly-brace sub-comp separator other than the crange TCL issue. Coding a custom TCL solution seems a bit over kill to me when there is no actual Cloverleaf bug to “work around”.

My 2 cents.

Ah… true… I was not looking at that resolution since there are so many places that that code needs to be added… I’ll meet with Peter and discuss adding this solution instead.  It will impact ALL HL7 financial messages, where I think I was looking at trying not to impact all the current state messages that work just fine  

Thanks!

Brenda 8)