Question: FRONT-END TPS TCL PROC PERFORMS 99% OF THE WORK

[Tom Parsons]

Hello,

Some of my interfaces with XLATEs have become rather unwieldy and would like to try a different approach…

I was wondering if there are any disadvantages in having an interface

use a single large TPS TCL proc to perform 99% of the work to

IDENTIFY/SELECT/MODIFY/CONTINUE/KILL inbound HL7 msgs,

located in either the INBOUND TPS or the ROUTE TPS

(with NO other TPS code and NO XLATEs).

Basically, all logic/data manipulation is then consolidated within a single proc.

So the “front-end” does all the work and just sends only the necessary

msgs thru to the outbound with their appropriate modifications.

As opposed to the typical / more “distributed method”-

1) INBOUND TPS code to IDENTIFY/SELECT (CONTINUE/KILL) msgs

2) ROUTE with TPS code and XLATE (MODIFY)

3) OUTBOUND TPS code to also CONTINUE/KILL msgs

My msg modifications (XLATEs) are many and require a significant amount of code-

(BULKCOPY/IF logic/TCL snippets/REGEXs/TABLE lookups/etc.),

along with a few scattered TPS procs.

The specific interface I have in mind runs thru ~5000 msgs daily.

But over the years these procs and XLATEs have become large and cumbersome to maintain.

Also, it is my understanding that large XLATEs especially pose a significant burden on system resources.

But any more so than large TPS TCL scripts ?

It seems to me that the more work you can do on the front end- the better.

Your thoughts ?

Thanks to all the CLOVERLEAF folks, GREAT product and forum !

Tom

[]

Topic

[Richard Hart]

Hi Tom.

We perform most of our translations in TCL. Previously management have blocked our request to share our code.  I’ve tried again and have a meeting later this week to discuss.

Essentially, we treat a message as string, split into segments and only parse the segments if a translation for the segments exists.

[Michael Hertel]

Hi Tom,

We don’t use XLATES at all here.

Everything is 100% tcl.

Either inbound or outbound as appropriate.

We made this decision well over 10 years ago for the exact reason you mention above. Not to mention the hassles of VARIANTS.

We don’t use 1 large tcl proc though, there are many but are named appropriately for each thread.

-mh

[Tom Parsons]

Hi Richard,

Thanks for trying to share !

I do already have TCL written to handle my interface.

TPS TCL procs are so much less of a hassle and as Michael mentioned-

no variants.

Hi Michael,

Thanks for your insight !

Yes, I should have done 100% TCL long ago too but old habits die hard.

I do try to build multiple procs for later code-reuse wherever I can.

And I should be able to hack the main pretty easily for my other interfaces.

Thank you both !

Tom

[]

[Richard Hart]

Hi Tom.

I haven’t received an update … but I have attached a sample of a Proof Of Concept translation.

The namespace setup

[Brandon Grudt]

I’m about 95% tcl here, with only a few remnant xlates that I haven’t had the urge to convert.  I’ve found my best guide to using tcl is to create a set of global tcl scripts that would find use in many scenarios (pass specific segments, kill by sub value, overwrite field, etc).  If I cannot pass a message correctly by using 2-3 of these standard scripts, I’ll create a specific one for that unique feed.  I’d be interested to hear what other sites do to handle this.

[Jim Kosloskey]

We are the opposite in that we are ALL Xlate and some Tcl where needed to extend the engine.

But I don’t think there is any Tcl which is responsible for transformation.

The Tcl we do have is largely but not completely generic in nature driven by arguments and re-used many times.

This has worked well for us.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[David Barr]

I don’t think storing message data in global variables is a good idea. What happens if your engine process is stopped with messages that have already been through inbound TPS but are still in the recovery DB?

I store that kind of stuff in message metadata.

[Max Drown (Infor)]

Except in very rare cases, it is not recommended to use Tcl instead of Xlates for parsing and mapping. We generally recommend that tcl be used for pre-xlate filtering and complex message transformations that cannot naturally be done in xlates.

We also do not recommend a pure Xlate environment. Instead, it is best to use both Tcl and Xlates in the manner that they are intended to be used, leveraging the power of both tool sets.

The xlate provides parsing and mapping functionality. Xlates are easier to develop and to support, especially for non-programmers. You can bring on new staff and get them trained faster doing xlate development than you can teaching them how to program or only hiring programmers.

There is a common misconception that xlates are slow. Xlates are relatively slower than tcl, but they are not slow. There are a lot of trade offs for switching to pure tcl other than performance that may not be noticeable for normal volumes of messages.

-- Max Drown (Infor)

[Richard Hart]

David Barr wrote:

I don’t think storing message data in global variables is a good idea. What happens if your engine process is stopped with messages that have already been through inbound TPS but are still in the recovery DB?

I store that kind of stuff in message metadata.

Hi David.

Cloverleaf keeps track of the module stack and the message has either been stored successfully after translation or not. If not, it is re-processed.

We use metadata to pass data between unrelated modules.

The globals are available for a single message translation. The namespace and access function (in this code Glob$HciConnName) allows a simple method to pass data (eg parsing characters) to the library routines or other translation modules.

Code:

   set myId 3
   set myField [lindex $aAllFields $myId]
   set myFac [RetComp Glob$HciConnName $myField 4]
   set myECode [EstablishmentList $myFac]
   set myField [ChangeField Glob$HciConnName $myField 4 R $myECode]
   set myField [RangeChk Glob$HciConnName $myField {20 8 8 4 1}]
   set aAllFields [lreplace $aAllFields $myId $myId $myField]

In this snippet, field 3 is the ‘working field’ and the library routines are:

 RetComp returns the 4th component of the field which is used  as a lookup value

 ChangeField Replaces the 4th component of the field with the new value

 RangeChk truncates the components to match the destination

Here is another ‘routing code’ snippet, where data is stored in globals and then logic applied:

Code:

   XlateMsg GlobRCTI$HciConnName

       #
       # Default the message routing to the ‘kil’ thread
       #
   set myTrxId KILL

       #
       # If the requesting/performing department is ‘XX’
       #     route to KILL (the default)
       #
   if {![string equal [GlobRCTI$HciConnName GET REQ_DEPT] XX] && ![string equal [GlobRCTI$HciConnName GET PERF_DEPT] XX]} {
           #
           # Default the message routing to the ‘SMAT’
           #
       set myTrxId SMAT

           #
           # If the requesting/performing department is ‘X’
           #     route to SMAT
       if {![string equal [GlobRCTI$HciConnName GET REQ_DEPT] X] && ![string equal [GlobRCTI$HciConnName GET PERF_DEPT] X]} {
               #
               # Check in the priority order and break if we have a valid
               # site
               #
           foreach myGlob {ACCT_NO REQ_DEPT PERF_DEPT} {
               set myCode [GlobRCTI$HciConnName GET $myGlob]
               set myWanCode [EstablishmentCode [EstablishmentCode $myCode S] wan ]
               if {$myWanCode ne “UNKNOWN” && [lcontain [GlobRCTI$HciConnName GET TRX_LIST] $myCode]} {
                   set myTrxId $myCode
                   break
               }
           }
       }
   }
   return $myTrxId
}

Please read Max’s post for Infor recommendations!

I started use TCL when we had isssues with version 3 and have found it very efficient to develop in – we have always been on *ix and work on the sever which also makes lie easier for us.

TCL translations work for us, but they do require a solid coding background!

[David Barr]

Richard Hart wrote:

David Barr wrote:

I don’t think storing message data in global variables is a good idea. What happens if your engine process is stopped with messages that have already been through inbound TPS but are still in the recovery DB?

I store that kind of stuff in message metadata.

Hi David.

Cloverleaf keeps track of the module stack and the message has either been stored successfully after translation or not. If not, it is re-processed.

We use metadata to pass data between unrelated modules.

The globals are available for a single message translation. The namespace and access function (in this code Glob$HciConnName) allows a simple method to pass data (eg parsing characters) to the library routines or other translation modules.

So you’re saying that the globals are only to pass data between TPS procs that are all on the inbound TPS stack? I was more worried about using the global variables to pass data from the inbound TPS to pre-xlate or OB TPS.

[Richard Hart]

David Barr wrote:

So you’re saying that the globals are only to pass data between TPS procs that are all on the inbound TPS stack? I was more worried about using the global variables to pass data from the inbound TPS to pre-xlate or OB TPS.

Typically the global variables are within the module to store data for later use or used in another ‘Xlate’ procedure within the module.

eg

In email creation, data is stored in the global variables during the ‘translation’ section and then used to create the email.

In this code snippet, data is stored in the various globals BLOCK_PT_IDENTIFIER, BLOCK_PT_VISIT .. and the mail template REPORT_EMAIL

Code:

               #
               # Send the email and a add debug display
               #
           if {$myEmail} {
               set myAddress [GlobPCSB$HciConnName GET REPORT_MAILUSER]
               set myRptPatientId [GlobPCSB$HciConnName GET BLOCK_PT_IDENTIFIER]
               set myRptPatientVisit [GlobPCSB$HciConnName GET BLOCK_PT_VISIT]
               set myRptMsgControlId $myMsgCtrlId
               set myRptMsgTS [GlobPCSB$HciConnName GET MSG_TS]
               set myRptMsgType [GlobPCSB$HciConnName GET MSG_TYPE]
               set myRptFacility $mySndFac

                   #
                   # Note that each segment is prefixed with a space so that the MS email
                   # format looks better.
                   # Otherwise each segment data line contains the segment ID!!
                   #
               set myRptMsg ” [msgget $aMsgId]”
               set myRptMsg [regsub -all “r” $myRptMsg “rn “]

                   #
                   # Substitute the values
                   #
               set mySubject  [subst -nocommands -nobackslash $mySubject]
               set myMessage  [subst -nocommands -nobackslash [GlobPCSB$HciConnName GET REPORT_EMAIL]]

               libEmail GlobPCSB$HciConnName $myAddress $mySubject $myMessage

               Print GlobPCSB$HciConnName “EMAIL TO     :>$myAddress$mySubjectn$myMessage<" I
           }

[Charlie Bursell]

First of all I hope we all know that you cannot pass globals from IB procs to Xlate or OB procs, and vice versa,  since they are in different interpreters.

Second, I think you should listen to Max as he is providing sound advice.

Why do you think we came up with the intergration engine in the first place?  It was so one could make easy GUI based changes and not have to go through code regression and all that it involves.

There is not a bigger proponent of TCL than me.  However, I would *NEVER* bypass all of the configuration options offered by Cloverleaf and other engines and do it all in TCL.

If you are going to do it all in TCL you are wasting your money on Cloverleaf or any other engine.  TCL is free!  You can even implement all your interfaces, TCP/IP, HTTP, etc. in TCL.  Do you want to do that?

Just  $0.02 worth from an old retired guy  😀

[Mark Thompson]

If you want data to be “global” as it moves through the various stages of life, store it in the message metadata – USERDATA field.  Best practice is to maintain USERDATA as a keyed list.  In a TPS proc storing the value “abc” in a key named MYVAR looks like this:

Code:

set userdata [msgmetaget $mh USERDATA]
keylset userdata MYVAR “abc”
msgmetaset $mh USERDATA $userdata

You can retrieve the value later in message processing with:

Code:

set userdata [msgmetaget $mh USERDATA]
keylget userdata MYVAR myvar
echo myvar=$myvar

If you want to protect against collisions with other USERDATA keys (or organize your keys), you can use a nested keyed list:

Code:

set userdata [msgmetaget $mh USERDATA]
keylset userdata MYGLOBALS.MYVAR “abc”
msgmetaset $mh USERDATA $userdata

and retrieve with:

Code:

set userdata [msgmetaget $mh USERDATA]
keylget userdata MYGLOBALS.MYVAR myvar]
echo myvar=$myvar

Note: Some early versions of Tcl did not handle keylset with values containing a period (.) as part of the data.  I belive that was fixed around Cloverleaf version 5.1.

- Mark Thompson
HealthPartners

[Mike Strout]

Interesting discussion. I have to admit that I was more than a bit shocked to hear that someone would intentionally not use Xlates for simple translations and mapping.

I am not saying that it is bad or ill-advised. I just can’t help but wonder if the performance gains are so significant that it is worth all the work coding in Tcl. It would be interesting to create a typical Xlate and a Tcl script that performs the same translations and run 100k messages through both to see just how much faster one is than the other.

I also wish we got enough traffic here to make poll on the topic worthwhile.

[Tom Parsons]

Hello All,

To be clear, re my post from 3/9/15…

Actually, I was just referring to 3 of my interfaces (out of 50+) that

I should have coded 100% TCL from the beginning, back in 2010.  

All of them have a common source/destination and are manipulated similarly.

These 3 are extremely complex (in my opinion) with complicated

msg filtering, modifications, and transformations.

Plus I wanted to “externalize” a lot of the future/anticipated msg-handling to be more “table-driven”, so I do not have to dive back into the code in

6 months for a minor change.

And btw, so far it has worked out well in production (past 2 weeks) even though I’m still on an old godforsaken version (CL 5.4 w TCL 8.3).  

Hoping to upgrade very soon.

Yes, I agree, it does not make sense to go 100% TCL for ALL interfaces, especially for those that can be coded/tested/implemented in a day or two with just a page-full of XLATE code and a couple of lightweight “tps procs” for filtering.

And taking advantage of the built-in development components and GUI to “hold your hand” is definitely worthwhile (even on old versions).

But for those rare “outlier cases”, PURE TCL does make a lot of sense.  

And whether all that complex-code resides in a couple XLATEs or in a

single/stand-alone TCL proc does not really matter that much (IMO).

You will still need the appropriate programming staff to support it.

BUT BEWARE, NOT FOR THE FAINT-OF-HEART !

Raw coding of a complex “pure TCL” interface requires a thorough understanding of the language and CL commands.

Plus it requires LOTS of time testing/debugging.

You must run thru all the code paths to make sure it works the way you expect and has no typos.  

Typos are incredibly easy to make and can be extremely subtle.

Runtime is the only time you can validate your code.  

Suggestions-

Test snippets of code at a time if possible

(always copy/paste back and forth).

Use TCL command line and “mini offiline-procs” to play/test with.

Build separate TCL procs (subroutines) when you can (reusable code).

Use REGEXs wherever you can.  

Test REGEXs for false positives.

If you don’t know REGEX, learn it !

Quite frankly, if you don’t know REGEXs, you probably should not be

writing a “pure TCL” interface to begin with.

Final suggestion-

Use LOTS AND LOTS AND LOTS of msgs during the testing process.  

TEST TEST TEST.

You will be glad you did.

Thanks to all responders !

Tom

[]

[Charlie Bursell]

Thanks for clarifying Tom.  That makes perfect sense.  Also some good advice.

In unique circumstances I too have had to do many complete interfaces in Tcl.  Going so far as using the low-level FRIO and XIO commands of the Cloverleaf library and calling outside applications from Tcl, i.e., cURL in the days prior to TclCurl and even sometimes a Perl or Java script.

Most of my early X.12 and web interfacse were entirely Tcl.

For the nay sayers of Tcl it brings up a point.  With a combination of Tcl and Cloverleaf I have *NEVER* encountered an interface we could not do!  Many have said we could not do it but we did.  And, I have done a heck of a lot of interfaces.

Tcl with Cloverleaf is not a weakness as many have said, it is the strenghth of Cloverleaf.

[Ronald Ortiz]

Perhaps if you did away with the Bulkcopy it may be easier to route only specific messages.  Are you also doing “_HCI_static_route_” for your routes?

If so, I would try changing that to something like this example below.

For example, to send ORU’s, setup as follows:

Quote:

Route: ORU_R01

Destination: to_cerner_oru

PreProcs: If you need an initial filter

xlate: Mostly Copy or Pathcopies (No bulkcopies at all)

Post Proc: If you really need one

Doing everything in TCL kinda takes away the magic of Cloverleaf to me because you don’t see the arrows showing the flow.  

What kind of messages are you receiving on that particular inbound thread?

[Author]

Replies