Multiple lined variable length record to single hl7 message

[D Huber]

I have been searching this site for a few days for a solution to my problem.  I haven’t been able to find a way to do this correctly yet.  I am fairly new to Cloverleaf.  We are using version 5.6PRev2.

I’m taking a current process and converting it to use Cloverleaf.  Our other interface engine can do this, so I know that Cloverleaf should be able to handle it. I’ve attached my documents that I’m using to test with.  No real PHI is included.  The actual file is much larger.

I need to use Cloverleaf to read a multi line file of variable record length and create an outgoing HL7 message.  So one file will have all the data needed to create the HL7 messege.  I’ve tried creating a VRL of a single field and adding it to a HRL that repeats.  I don’t know how many lines are in the file, so I’m not sure how to define the repeating parameters.  Just using the tester, I’m unable to get more than the first line to come back.  I set the tester to use EOF Terminated and process all records.  I also hard coded the repeat to be 5 just to get something to work.  However, none of this works.  You can see that the first several lines have data in <> tags.  I will need to parse through this in the HL7 translation at some point.  I obviously will also be doing a lot of hard coding in the translation to create a full messege.  The data starting with DISCHARGE SUMMARY will be the contents of the OBX.5.

So where do I start?  I’ve run out of options just looking through this forum.  Am I going at this totally wrong?  Any advise is greatly appreciated.  Thank you.

Topic

[Robert Kersemakers]

The way I would do it is like this:

* Read the file as ‘single’, so the file is seen as one message.

* Make a tcl-proc to parse through this message. Take out the first tagged fields and strip off the actual value. Then gather the complete summary together, where every (non-empty) line is concatenated by a ~. Make a new message, where you separate all these fields by a delimiter (say ‘;’). You will have something like . This is your VRL.

* Define this VRL and use it to build a translation from this VRL to HL7.

The hardest part obviously is the tcl-proc. Hope this helps to give you an idea of how to do it.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[D Huber]

Robert

Thank you for the reply.  I was thinking I would probably have to dig into TCL to get this to work the way I need it to.  Is there a thread that you know of that has an example of how to read a file in as a ‘single’ and to pass it to a TCL proc?  If not, could you give me some direction on how to do that.  I have seen it mentioned a few times in the forums, but I still haven’t been able to figure out how it is done.  

Also, will the testing tool allow me to set this up for testing, or is there another way I will have to go in order to test my progress?  I will do some more searching today to find these answers, but if you (or another tech) could help guide me, I would really appreciate it.  This has been the hardest task to figure out since working with Cloverleaf.  There just isn’t a lot of information about it that I can find.  Thanks again for your help.

[D Huber]

I think I’m starting to get the puzzle pieces in place.  In the NetConfig I would create a thread.  In the thread properties I would set the Protocol to PROTOCOL:fileset-local.  Under the protocol properties I would set the Inbound Style to ‘Single’.  I then would output that to the tcl proc.  I’m still searching on how to get the tcl proc setup.

Would this read in any file under that Inbound directory?  Wouldn’t I need to then move/mark the files that have already been processed?  That part will be later, but I’m starting to wonder how I’m going to do that as well.

I thank you again for any advice with this.  I will keep posting back what I find in order to make sure I’m still on track and to document how to do this for future Cloverleaf users that find themselves stuggling with a similar issue.

[Robert Kersemakers]

Reading a file as ‘single’ is very easy. In the inbound thread, choose protocol ‘Fileset Local’; in the Protocol Properties you can then choose style ‘single’. The contents of each file will then be read as a single message. The file will then be deleted.

Then you will need to build a tcl-proc to put in TPS Inbound Data in the same thread. Here is an example of a likewise proc:

Code:

######################################################################
# Name: orbis_verwerk_TIS_facturen
# Purpose: het verwerken van een TIS facturenbestand
# UPoC type: tps
# Args: tps keyedlist containing the following keys:
#       MODE    run mode (”start”, “run” or “time”)
#       MSGID   message handle
#       ARGS    user-supplied arguments: Geen
#              
#
# Returns: tps disposition list:
#

proc orbis_verwerk_TIS_facturen { args } {
   keylget args MODE mode               ;# Fetch mode
   set module “orbis_verwerk_TIS_facturen”;
   keylget args ARGS uargs;
   set debug 0  ; keylget uargs DEBUG debug;
   
   set dispList {} ;# Nothing to return

   switch -exact — $mode {
       start {
           # Perform special init functions
   # N.B.: there may or may not be a MSGID key in args
       }

       run {
          # ‘run’ mode always has a MSGID; fetch and process it
          keylget args MSGID mh

          set msg [msgget $mh];
          lappend dispList “KILL $mh”
          set newmsg {}
          regsub -all n $msg “” msg
          set linelist [split $msg r]
          set newlist {}

          # Eerst alle regels doorlopen om niet benodigde regels eruit te gooien.
          foreach line $linelist {
            # Bepaal eerst rekeningnr, dagboek en boekstuk
            set rekeningnr [string trimleft [string range $line 84 94] 0]
            set dagboek [string trimleft [string range $line 9 11] 0]
            set boekstuk [string trimleft [string range $line 34 42] 0]
           
            if {$debug > 2} {
              echo “rekeningnr: ; dagboek: ; boekstuk ”
            }
            # Bepalen of regel verwerkt moet worden: alleen als rekeningnr gevuld is en dagboek = 46.
            if {[string length $rekeningnr] > 0 && [string equal $dagboek “46”]} {
              lappend newlist [list $boekstuk $line]
            }
          }
          # De opgebouwde lijst sorteren op de eerste index (boekstuk)
          set newlist [lsort -integer -index 0 $newlist]

          # De nieuwe lijst verder verwerken en opsplitsen in Header en Detail regels
          set totaal_header 0
          set first_header “X”
          set boekstuk_vorig “”
          foreach element $newlist {
            lassign $element boekstuk line
            if {$debug > 2} {
              echo “boekstuk: ; line ”
            }

            # >>> Maak eerst de DETAILregel aan. <<<
            # Referentie
            set referentie [string range "${boekstuk}                " 0 15]
            # Grootboekrekening
            set rekeningnr [string trimleft [string range $line 84 94] 0]
            set grootboek [hcitbllookup TIS_rekeningxSAP_grootboekrekening.tbl $rekeningnr]
            if {[string equal $grootboek "XXXgeenXXX"]} {
              set grootboek ""
            }
            set grootboek [string range "${grootboek}          " 0 9]
            # Kostenplaats, Bedrijfsnummer en Ordernummer
            set kostenplaats_tis [string trimleft [string range $line 288 294] 0]
            set bedrijfsnr_tis [string trimleft [string range $line 0 4] 0]
            # Kostenplaats opzoeken in tabel, waarbij nieuwe kostenplaats, bedrijfsnummer en eventueel ordernummer worden gegeven
            set result [hcitbllookup TIS_kostenplXSAP_kostenpl_bedrijfsnr.tbl $kostenplaats_tis]
            if {[string equal $result "XXXgeenXXX"]} {
              if {[string length [string trim [string trim $ kostenplaats_tis 0]]] 99} {
              set bedrag “[string range $centen 0 end-2],[string range $centen end-1 end]”
            } else {
              set centen [string range “00${centen}” end-1 end]
              set bedrag “0,${centen}”
            }
            if {[string equal $negative “X”]} {
              set bedrag [string range “${bedrag}                ” 0 15]
            } else {
              set bedrag [string range “-${bedrag}                ” 0 15]
            }
            # Regel compleet
            set detail_regel “${referentie}t${grootboek}t${bedrijfsnr}t${bedrag}t${kostenplaats}t${ordernr}r”
            set detail_mh [msgcreate $detail_regel];
            set driverctl “{FILESET {{OBFILE DETAIL.txt}}}”
            msgmetaset $detail_mh DRIVERCTL $driverctl
            lappend dispList “CONTINUE $detail_mh”
            if {$debug > 1} {
              echo “detail_regel: ; totaal_header: ”
            }

            # Nagaan of Header-regel moet worden aangelegd
            if {[string equal $first_header “X”] || [string equal $boekstuk $boekstuk_vorig]} {
              set first_header “”
            } else {

              # >>> Maak de HEADERregel aan. << 99} {
                set bedrag “[string range $centen 0 end-2],[string range $centen end-1 end]”
              } else {
                set centen [string range “00${centen}” end-1 end]
                set bedrag “0,${centen}”
              }
              if {[string equal $negative “X”]} {
                set bedrag [string range “-${bedrag}                ” 0 15]
              } else {
                set bedrag [string range “${bedrag}                ” 0 15]
              }
              # Omschrijving
              set omschrijving [string trimleft [string range $line 54 83] 0]
              # Regel compleet
              set header_regel “${bedrijfsnr}t${debiteurnr}t${factuurdatum}t${boekdatum}t${referentie}t${bedrag}t${omschrijving}r”
              set header_mh [msgcreate $header_regel];
              set driverctl “{FILESET {{OBFILE HEADER.txt}}}”
              msgmetaset $header_mh DRIVERCTL $driverctl
              lappend dispList “CONTINUE $header_mh”
              if {$debug > 1} {
                echo “header_regel: ”
              }
              set totaal_header 0
            }
          }

     }
    }
   return $dispList
}

Just use this proc as reference; especially the beginning of the proc (in ‘run’ mode) will be interesting for you.

I see you found out the first part yourself already.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Robert Kersemakers]

To start a (new) proc, just open the Script Editor. You can then open an existing file with procs, or start a complete new file.

You could make a copy of my proc, put it in the file and then change the name of this proc (the name is now ‘orbis_verwerk_TIS_facturen’).

Please ignore all Dutch comment… 😉

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[D Huber]

Robert

Thank you very much for your help.  The pieces are finally coming together.  I will work on this today.  The response one this forum is actually amazing.  Thanks again.

[D Huber]

I got it all to work finally.

[Robert Kersemakers]

Glad to hear you got it all working.

Echos are really necessary when testing your tcl-proc. Don’t forget to remove them after testing, or they will flood your log-file. Or use a parameter to show the echos if you have the need for it. It saves a lot of work when you have to debug this proc again in the future.

Making a copy of the output file in another directory does take a bit of work/performance; not just a switch to turn on. You will need another outbound thread that writes the messages to a different directory/file. Then route the messages from the inbound thread to both outbound threads (original and copy), of course after Xlate. You can do this with only one route: chose both outbound threads as destination.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Jim Kosloskey]

D,

If your source system does not want their files deleted from their directory after you pick them up, you will need to insert some Tcl at the Fileset/FTP TPS deletion UPoC. That Tcl will remove the file(s) from the deletion list so Cloverleaf does not issue a delete.

HOWEVER – someone has to delete those files before the next processing cycle or they will be multiply processed – not usually a good thing.

If the purpose of an additinal route is to make a COPY of the inbound file(s) then that route should not have an Xlate but rather be raw routed.

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

[Robert Kersemakers]

Oops, I misread: you want the original files saved as a copy.

Another option: SMAT will have a copy of the inbound messages/file. But most likely the users will want something they can ‘use’ themselves.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[D Huber]

In my proc I am adding code to just save the raw message to a file in another directory prior to any manipulation.

[Jim Kosloskey]

D,

In the Fileset Local protocol there is a place for a directory parse Tcl proc.

Att he point that UPoC gets invoked (your proc here) you are presented with a list of the file names that were found upon directory scan.

You can modify that list and return it to Cloverleaf and then Cloverleaf will process the files in that list.

At this UPoC (like most UPoCs) this is a Tcl stack so you can specify as many Tcl procs as you would like (in case you use a modular approach) to do the job you want.

For example I have procs that modify the list another that moves the files in the list, another that detects if the list is empty, etc.

In any of the procs at this UPoC you can do anything you want with the actual files. You have the file names but at least until 5.6 you do not have the directory name(s) where the files reside – that issue is relatively easy to resolve but will be much better when the directory names are provided as well.

Email me if you want to go into further depth.

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

[D Huber]

That worked great.  Thank you very much.

[D Huber]

Just when I thought I was done with this project, I found a new snag.

The files in the directory sometimes contain a space.  CL is unable to find the file, because it cuts it off at the space in the file list.  I hope this is the last question for this, but how do you handle spaces?

[Jim Kosloskey]

Is this Unix or Windows?

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

[Robert Milfajt]

There is a TCL command you could use in the dirParse routine to create your own file list.  Here is an example of how we use it.  I tested using hcitcl with a filename with a space and it added it to the list.

Code:

set dirlist [glob -nocomplain -directory $sourcedir *.doc]

The heavy lifting here is going to be how you integrate this into the script to return your own custom file list back to the Engine.  In my example, we are not running this in the Engine, but on a Windows server as a scheduled task.

Hope this helps,

Robert Milfajt
Northwestern Medicine
Chicago, IL

[Jim Kosloskey]

D,

Cloverelaf 5.6, Rev2 AIX.

I just tested this and the Fileset_local protocol found a file name with imbedded space for me.

If you have Tcl involved, make sure you are handling what the UPoC gives you as a LIST. If you do not you will only get partial names in your processing.

The list that Cloverleaf provides (at least with 5.6) is complete as far as I can tell.

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 61 years IT – old fart.

[D Huber]

Her is the server info.

Current Platform:

   Java version: 1.5.0_07

    Java vendor: Sun Microsystems Inc.

        OS type: Linux

     OS version: 2.6.18-92.el5

        OS arch: i386

Server Build Information:

     Version: 5.6PRev2

        Date: Mon Jun 2 2008

        Time: 02:42:09 PM

    Platform: Windows_NT

 Java Vendor: Sun Microsystems Inc.

 JDK Version: 1.5.0_07

Swing Version: 1.5.0_07

 RMI Version: 1.5.0_07

[Author]

Replies