Replacing HL7 characters with escape sequences…

[David Gordon]

I’m banging my head on my desk with this one…

We are converting an enormous number of records from three source systems for delivery to an HL7 system.  The issue is that these source systems (one sending VRL, one FRL, one XML) all contain multiple instances of ampersand, pipes, etc, in free text fields.  The vendor on the destination system wants these mapped to the HL7 escape sequences, for instance & to T.

This should be easy to do via a tps or xltp using regsub or string map, but there are several issues that arise.  When the tcl proc hands back the converted value, the slashes seem to cause a ton of problems.  Sometimes the slashes just vanish, sometimes they don’t.  Sometimes two characters back to back (i.e. &&) results in the escaped sequence (T\T\ in this case) to drop all but one of the slashes (T).  If the final character in the field is a special character, the proc will fail as it tries to wrap the field in braces, and it freaks out when it finds } at the end of the value.

There are estimated to be 200,000 records with these special characters in them so manual correction is not an option.  It seems to be an issue with either the engine or TCL not liking the use of the HL7 escape sequences.

Has anyone had any success with something like this?  How did you get it working?!

We are on 5.4.1 SP2 on AIX.

Thanks!

Topic

[Jim Kosloskey]

David,

Something to keep in mind is that the length of these fields and records just got longer as a result of this substitution.

For VRL, that may or may not be an issue.

For FRL that definitely can be an issue.

For XML that also may be an issue.

Depending on how you are doing your Xlate there could be truncation in the HL/7 lengths.

I am not sure that is the total answer to what you are seeing but it certainly could be a part of the total answer.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[David Gordon]

Hi Jim,

I took that into account, and it’s definitely not a length issue.  I can have && as the only values in a field, and it still comes back to the engine as TT

Honestly there seems to be another layer of evaluation when TCL hands the string back the engine, and it freaks out whenever the back slashes are involved.

[Jim Kosloskey]

David,

One more thing the is a Tcl escape character so things like } may result in }.

So you may need to escape the HL/7 escape character (llooks weird when the HL/7 escape character is also the Tcl escape character) – something like \T\.

Also with the map command I sometimes find using a list for the map specification works better. Something like this:

string map

$variable

Also if you set a varaible up as your pattern for replacement like this:

set hl7esc “\”  <– makes hl7esc =

append junk $hl7esc T $hl7esc <– makes junk = T

string map

$variable < should replace all & with T.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[David Gordon]

I’ve tried all these suggestions with no luck.

It works in *most* cases, but not all.  It seems as though there is another layer of evaluation that happens when TCL passes the string back to the engine and it can’t handle the HL7 escape sequences.

Interestingly enough, the TCL proc contains the correct value, it’s just when the translation is run the message does not contain what TCL is returning.

I *can* get the message to show the correct escape sequences if I return the values by wrapping them in a list command (i.e.  set XlateOutVals

) but this introduces braces in values that have a space in them, which then shows up in the message.

There has to be another method that will work here.  My code at the moment is:

Code:

proc xltp_ReplaceContCharInline {} {
   upvar xlateId       xlateId
 xlateInList   xlateInList
 xlateInTypes  xlateInTypes
 xlateInVals   xlateInVals
 xlateOutList  xlateOutList
 xlateOutTypes xlateOutTypes
 xlateOutVals  xlateOutVals

set inRecords $xlateInVals

puts “VAL IN $inRecords”
# Assign all the control characters and escape sequences to variables
# This step was suggest by Lawson to help with some unusual results from the proc.

set HL7Esc “\”
append newAmp $HL7Esc T $HL7Esc
append newSlash $HL7Esc E $HL7Esc
append newPipe $HL7Esc F $HL7Esc
append newCarrot $HL7Esc S $HL7Esc
append newTilde $HL7Esc R $HL7Esc

# String map out all the control characters

set inRecords [string map [list & $newAmp \ $newSlash | $newPipe ^ $newCarrot ~ $newTilde] $inRecords]

puts “VAL OUT $inRecords”

set xlateOutVals $inRecords
}

In the event that the value passed is APO-BU&IPRONE, it should return APO-BUTIPRONE but I get APO-BUTIPRONE.  Oddly enough, if I send it APO-BUIPRONE it correctly sends back APO-BUEIPRONE.

This is incredibly frustrating.

[Charlie Bursell]

Seems a lot of code for a simple problem.  First thing is you rae not treating xlateInVals and xlateOutVals as lists!!!

set map

set xlateOutVals

]]

Note there are *NO* meta characters other than “” in the string map command so only the “” need be escaped

[David Gordon]

Well the code I have works, but Cloverleaf somehow modifies the correct, returned string from TCL.

I’ll try yours and see if it improves at all, Charlie.

[Charlie Bursell]

Cloverleaf is doing exactly what you told it to do  😀

[David Gordon]

Hmm, okay this seems to be working now, although I used only \ instead of \\ before and after in the escape sequences.

Can you explain how this is any different than what I had before?  Is it simply returning it in a list that makes it work?  When I did that before anything with a space in it was returned wrapped in {} but this doesn’t do that…

[Charlie Bursell]

Mainly I treated the values xlateInVals and xlateOutVals as *LISTS*

One of the most common mistakes when writing xlate code

[David Gordon]

I had tried using a list on the outbound, but I don’t think I ever did the same on the inbound.

You’re the man Charlie.  If you get a chance to come back to Winnipeg, I owe you a beer.

[Max Drown (Infor)]

https://usspvlclovertch2.infor.com/viewtopic.php?t=3908&highlight=escape

-- Max Drown (Infor)

[Author]

Replies