sub component separator fails Xlate when changed by sender

[Peter Heggie]

Cloverleaf 6.2.0.2 – When the sending system includes an ampersand in the message text, it changes the sub component separator from an ampersand & to a curly bracket {.

When the translator has a COPY action that uses a source which includes the curly bracket {, it fails. But it works fine if the message did not have an embedded ampersand and the MSH delimiters were never changed and the sub component separator is an ampersand.

Full disclosure – we were not specifying the sub component. But we never had to. Here is the source portion of the COPY statement:

6(%g1).IN1(0).#16(%f1).[9]

This has always worked, even if there was a sub component in sub field 9, as long as the delimiter was an ampersand. But if the delimiter has changed to a curly bracket, then it fails.

If we change the translator to specify the sub component, it works. But I have done just a few tests and am not ready to throw in this change on the day before a long holiday, so we will test more next week.

So should Cloverleaf handle the curly bracket, or have we been skating along on thin ice all this time?

6(%g1).IN1(0).#16(%f1).[9].[0] – this works

Peter

Peter Heggie

Topic

[Michael Hertel]

My two cents:

It’s really not changing it to a curly bracket, it’s instead returning a list.

{$KEY $VALUE}

It’s picking up the first character which is the curly bracket.

[Peter Heggie]

I’m probably not understanding – the source system is sending us a message with a curly bracket as a delimiter, And there is a curly bracket in the incoming message. Cloverleaf should recognize this as delimiting a sub component, but it is thinking it is a list.

Peter Heggie

[Jim Kosloskey]

What is in the sub-component separator position of MSH-2 in the message from the sending system?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Michael Hertel]

I misunderstood.

I did not realize the curly bracket was being done by the source.

I read the post as you received the ampersand and the Xlate was returning the curley bracket.

My bad. Nevermind.

😳

[Peter Heggie]

sure no problem

the incoming message MSH is this:

MSH|^~{|SOARF|N0A8|||201711212016||ADT^A08|e46a654e-40fd-480f-8257-94b8f549bf3a|P|2.7||1

the incoming message IN1 segment is this:

IN1|1|151^Excellus HMO|546^^^900000^XX^^20140904~NYC01000^^^184^CRI^^XXXXXXXX~XXXXXXXXXX^^^900001^TX^^XXXXXXXX|Excellus Blue Cross Blue Shield^L^546^^^900000^XX|PO BOX 21146^^Eagan^MN^55121^USA^M^^^^POLCS-M~PO BOX 22999^^ROCHESTER^NY^14692^USA^M^^^^PMain~PO BOX 21146^^Eagan^MN^55121^USA^M^^^^HRefund|^^^^^^^POLCS-M|^WPN^PH^^0^800^9208889^^POLCS-M~^WPN^PH^^0^800^9198810^^PMain~^WPN^PH^^0^800^9208889^^HRefund|XXXXXXXX|Excellus HMO|XXXXX^^^900000^XX^^XXXXXXXX|Self Employed^L^XXXXX^^^900000^XX||||Health|XXXXXXXX^XXXXX^X^^^^L^^^20171019~XXXXXXXX^XXXXX^X^^^^L^^^20160520{20171019|Self|XXXXXXXX|…………

Peter Heggie

[Jim Kosloskey]

So if you populate the OB MSH-2 with the { for the sub-component separator it works?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Peter Heggie]

no, I meant to say if I edit the message and replace the { with a & in MSH2, and find the spot in the message that had an ampersand and replace it with some innocuous character like an underscore, then the presence of the ampersand in the IN1 segment does not cause a problem. It is treated as a sub component separator, which it is.

But the other method which works, is if I leave the { in MSH2 as the sub component separator, and then modify the COPY statement to specify the subcomponent like this

example a:

6(%g1).IN1(0).#16(%f1).[9].[0]

instead of our original code:

example b:

6(%g1).IN1(0).#16(%f1).[9]

then the translate works also.

This leads me to believe that Cloverleaf is working correctly (when the COPY statement specifies the sub component as in example a) and we have been depending on ‘beneficial’ behavior which actually should not be relied upon if there are sub components with certain delimiters.

Maybe it is sort of like putting code in the COPY action that does a set command to get the value of $xlateInVals – if you use

set indata $xlateInVals vs. set indata [lindex $xlateInVals 0]

You really can’t depend on the first method. Similarly, if your data has sub components, or might have sub components, we should probably specify the sub component in the COPY action. We have been ‘getting away with’ vague coding because the sub component delimiters are usually ampersands.

Peter Heggie

[Jim Kosloskey]

So what ‘should’ happen IMO is Cloverleaf should change the incoming { to an & on output (since HL/7 specifies that component of that field having subcomponents) unless you change the OB MSH-2 inside the Xlate.

Now I know post 5.8.5 Cloverleaf was modified a number of times to more properly handle sub-components. During various versions of those changes there was no need to specify sub-component notation during a COPY (as I recall that kept changing and I do not know what was finally settled on). PATHCOPY on the field seemed to work most of the time in most of the releases. Sub-component notation always worked and appears to still work.

Also I think your past success probably stemmed from the fact that the variants default to the standard for the various separators.

What exactly is the ‘failure’ you are experiencing?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Peter Heggie]

QDX_Tcl_SplitList({2017101) failed: unmatched open brace in list

[xlt :xlat:ERR /0:financialin_xlate:11/22/2017 08:15:40] [0.0.59578574] Xlate ‘sf_ADT.xlt’ failed: Internal XPM Tcl error

[xlt :xlat:ERR /0:financialin_xlate:–/–/—- –:–:–] QDX_Tcl_SplitList failed:: unmatched open brace in list

errorCode: TCL LOOKUP COMMAND absolutely_ridiculous_command

[xlt :xlat:ERR /0:financialin_xlate:–/–/—- –:–:–] errorInfo:

[xlt :xlat:ERR /0:financialin_xlate:–/–/—- –:–:–] invalid command name “absolutely_ridiculous_command”

[xlt :xlat:ERR /0:financialin_xlate:–/–/—- –:–:–]     while executing

[xlt :xlat:ERR /0:financialin_xlate:–/–/—- –:–:–] “absolutely_ridiculous_command”

[xlt :xlat:ERR /0:financialin_xlate:–/–/—- –:–:–]

Peter Heggie

[Jim Kosloskey]

Is there a Tcl proc being used in conjunction with the COPY or is the QDX_Tcl_SplitList proc not one your shop invokes?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

Personally if I know there are sub-components for a component I use subcomponent notation or a PATHCOPY for the component (hopefully that still works in 6.2).

I go that way basically because in CL 3.1 through 5.6 subcomponents were kind of not handled uniformly by CL but the COPY with sub-component notation did work and once PATHCOPY was introduced that would also work.

However, as long as Infor is trying to make the sub-component handling better it is my position when copying a field with components which is defined in the standard as containing sub-components, the sub-component separators should be handled appropriately without requiring sub-component notation at the field’s component level.

My questions regarding setting of MSH-2 is my attempt to find out if the Xlate was sensitive if the ob sub-component separator is set within the Xlate in 6.2 perhaps as a method to assist.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Peter Heggie]

I thought,that,was,a,Cloverleaf utility. We do not have a TCL proc or code in that copy action.

I fh8nk we should spell out the sub component in the source. We only wa,t the 0th sub component so a path copy is ,of ideal.

Peter Heggie

[Jim Kosloskey]

You are probably correct that the failing Tcl proc belongs to Infor.

IMO this should not happen. If you are so inclined, you could report this as a bug.  

I would but I don’t have Cloverleaf or Xtreme access right now.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Rob Abbott]

as Jim says this looks like a bug, please open a case in Xtreme so we can get it addressed.  thanks!

Rob Abbott
Cloverleaf Emeritus

[Kimberly Drew]

Hello – just checking to see if anyone has an update as to when INFOR may put out a patch for this bug.  thanks

[Peter Heggie]

Support is checking with Development. I’ll post any response.

Peter Heggie

[Steve Williams]

Working with Peter offline, I was able to reproduce the issue and Michael Hertel was dead on to the answer; although he may not have known it.

In the Copy stmt you are sending in two dates separated by a curly brace and then treating the entire component as a scalar value. In trying to extract the second date with the crange fn based on the assumption of the values length, you create the error. What is being missed in this case is the { is a reserved character in TCL. Cloverleaf knows this and escapes it for you with a backslash; Cloverleaf has many hidden jewels like this. This makes the subcomp separator two chars long in the xlateInVals data set.

When you run the crange function, your target text is actually “20160520{20171019” and the return text becomes “{2017101”, not what you have expected. This is in turn is passed into xlateOutVals, which makes it a malformed TCL list, which throws the XPM error.

You can simulate the error with this code in any Xlate COPY stmt:

Code:

set test1 “20010428\{20010531”
echo “test1 = $test1”
set xlateOutVals [crange $test1 9 16]
echo testX = $xlateOutVals

[Brenda Carpenter]

Hi all,

Peter did open a ticket for this, but I also did find a work around to replace the ampersands with the word ‘AND’ and then replaced all the curlybraces with the ampersands.  

Initial testing looks good, I am going to do a compare and volume test before implementing, but this will put the message back to the way we expect it to normally be.  In example below, patient access team accidentally used the ampersand in the patient reason for visit field.

Before:

MSH|^~{|SOARF|J0A8|||201711011601||ADT^A08|

PV2|||^COLD SYMPTONS & EAR PAIN|||

GT1|1|….|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520{20170716||

IN1|1|194^POMCO|…|HEALTH|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520{20170716|SELF

Code used to revert the delimiter back to an ampersand:

       set curlybrace “{“

       set addlsubFieldSep [cindex $msg 7]       ;  #MSH2 input field

        if {“$addlsubFieldSep” eq “$curlybrace” } {

             regsub -all {[&]} $msg {AND} msg

             regsub -all {[{]} $msg {&} msg

Giving output as below:

MSH|^~&|SOARF|J0A8|

PV2|||^COLD SYMPTONS AND EAR PAIN||||~~~~FALSE|

GT1|1|…|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520&20170716||…

IN1|1|194^POMCO|….|||HEALTH|STONECOLD^OUT^^^^^L^^^20170716~STONECOLD^OUT^^^^^L^^^20160520&20170716|SELF|…

We are not actually using or passing the second date field downstream, we are merely checking for the presence of one to see if the data has a termination date.

Any questions, let me know  8)

Thanks.. Brenda

[Jim Kosloskey]

This is something I have seen over and over since the 70’s when HL/7 first came into being.

The original authors of the HL/7 standard wanted to be able to allow the use of the separator characters within text and not be treated as separators.

How they decided to do that was to specify an escape sequence for the various MSH-2 values. That is what the escape separator is for in the MSH-2.

The original framers of the HL/7 standard (I recollect this actually existing in pre 2.x) understood that any parser would have issues with spuriously placed separators.

This is all very clearly specified in the HL/7 specification. For most of the versions of 2.x it was in the same section of Chapter 2 (2.7 to be specific).

In later releases that was moved and a new separator was added but it is still there.

IMHO the ONLY way to handle this properly is for the sending system to apply the proper escaping when building the messages. ALL other ‘tricks’ deployed simply are waiting for failure. In the 40+ years I have worked with Integrating HL/7 I have NEVER seen a fool proof non-standard resolution.

It is also incumbent upon receiving systems to un-escape properly. Although that can be accomplished easily and consistently by Cloverleaf but then the improperly placed separators become a problem for the receiving system’s parser.

Specifically, for this workaround as an example, what happens if a user of the sending system inadvertently uses { when they wanted ( or [?

The work around will change the spurious { to a & and now that will appear to be a subcomponent where potentially non should exist.

There is no way other than deploying the standard to accommodate all the possible permutations.

I submit if you never want to be worried about spurious separators causing you issues then pressure needs to be brought on the sending system to follow the standard – a least this rather detailed and specific part of the standard.

It can be done. The last project I worked on at MDACC was an upgrade to the Lab system. I forced the vendor to follow this portion of the standard and we never had an issue after. I got pretty beat up and certainly did not make a friend in the vendor but I would do it again.There is no unmarked land mine laying there for the Integration engineers that followed me.

I apologize for wandering a bit off of the specific issue – but this is related and I think important.

Anyway that is my .02 (may even be worth less than that to many).

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Brenda Carpenter]

Thanks Jim for your opinion and words of advice, I greatly appreciate it!  

In my example, I am only changing the { to an & when one resides in the MSH2 as the  sub sub field delimiter instead of the place where the ampersand is – this would indicate that it is not used in any other field in the message.  Now in the case if they used non standard delimiters and they used the ampersand as a different qualifier in MSH2, then we would be in trouble – I should add additional code to make sure the other three delimiters are NOT an ampersand to ensure that doesn’t happen.  When dealing with humans (the patient access team), there is always a chance of error in freeform text fields and we cannot go to the vendor in that case – only re-iterate the importance of not using special characters when this issue arises.  In the other instance where this happened we did ask that vendor to NOT send the ampersands in the data/message for our messages.

We will continue to monitor this, thanks everyone for your inputs!

Brenda

[Steve Williams]

Brenda,

The HL7 message is not malformed. The sending system is using the curly brace as the sub-comp separator, which does follow the HL7 guidelines. Cloverleaf is supporting the curly brace as a sub-component separator as well. The only error I found here is the extraction of an HL7 component, the assumed length of the component field, the assumed presence of a single character sub-comp separator and the use of a TCL crange function leading to a malformed TCL list passed back via xlateOutVals.

The xlate solution to  this issue was already mentioned by Peter in a previous post. It will fix the issue and work 100% of the time. Bug free.

Peter:

Quote:

But the other method which works, is if I leave the { in MSH2 as the sub component separator, and then modify the COPY statement to specify the subcomponent like this

example a: 6(%g1).IN1(0).#16(%f1).[9].[0]

instead of our original code:

example b: 6(%g1).IN1(0).#16(%f1).[9]

then the translate works also.

This leads me to believe that Cloverleaf is working correctly (when the COPY statement specifies the sub component as in example a) and we have been depending on ‘beneficial’ behavior which actually should not be relied upon if there are sub components with certain delimiters.

In effect, use Cloverleaf’s COPY command with the sub-comp path to extact that second date value and all works as expected and desired, regardless of the sub-comp character used by the sending system.

I did not see any mention of other issues with the message and it’s curly-brace sub-comp separator other than the crange TCL issue. Coding a custom TCL solution seems a bit over kill to me when there is no actual Cloverleaf bug to “work around”.

My 2 cents.

[Brenda Carpenter]

Ah… true… I was not looking at that resolution since there are so many places that that code needs to be added… I’ll meet with Peter and discuss adding this solution instead.  It will impact ALL HL7 financial messages, where I think I was looking at trying not to impact all the current state messages that work just fine  

Thanks!

Brenda 8)

[Author]

Replies