Throttling messages inbound to Cloverleaf from a foreign sys

[John Bonnie]

Hello all,

I’m looking for a way to throttle messages inbound to Cloverleaf from a foreign system.  In this case I want an inbound thread to process only one hl7 tcp/ip message a second from a foreign system.  Any ideas for this?

Thanks in advance.

Topic

[Robert Kersemakers]

Hi John,

I wouldn’t try to throttle messages as they are being received by Cloverleaf. You could put in a ‘sleep 1’ or something similar at the moment you send back an ACK, which would take care of your problem. But this ‘sleep 1’ will stop the entire process! So not a good idea.

Normally you will do this because the receiving system isn’t capable to process a large amount of messages. So just receive the messages from your sending system and write each message as a file in a certain directory. Then use fileset-local to read this directory and only process 1 file per second.

You can even do this in 1 thread.

A -> B -> C

A the thread that receives the messages from your sending system.

B is a fileset-local thread that writes the messages to a directory (outbound). But B is also inbound, so it reads the same directory and sends a message to C. You can configure this in such a way that only one file/message per second is taken.

C is the thread sending the messages to the receiving system.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Terry Kellum]

I once did this with a sqlite table.  One process would put in the transaction with a time stamp.  The next process inline would search the table for transactions with timestamp less than (now – x).  The tricky part is that you need to also mind the transactions metadata.

I did this because our LIS would process orders quicker than ADTs, and we were generating ADTs from the order transactions.  The order would hit without a lillypad to sit on.  Slowing down the orders by 5 seconds allowed the ADT to hit first.

[Mark Thompson]

Terry,

Were you able to preserve the metadata – especially Original Message MID?

We were looking at switching our file based queues into a database, and hoping to preserve the metadata as a bonus.

- Mark Thompson
HealthPartners

[Terry Kellum]

I think I was dummying up the metadata.  I knew where it was going so there are only about three fields that you NEED if you’re not doing something complex.  I would think that you could save the metadata using msgmetaget.  You could then set the metadata on the newly (re) minted message using msgmetaset.  RTFM has some good data regarding the RO and RW fields.  (TCL Extensions).

Here’s the code that I used.  Keep in mind that this code may cross the streams, or result in causing our own sol to go supernova.  If you break your engine or messages with it, you own both pieces.

Code:

######################################################################
# Name: delay_msg.tcl
# Purpose: Provides delay of messages moving thru the engine.
#                Places the message into a sqlite database table stamped with the
#                tcl time (clock seconds).  In read_delay_msg the databse tables
#                is queried for messages that are old enough to be sent on.
#                Suggested use is in UPOC protocol.  “write” in outbound and
#                “read” in inbound.
#
# UPoC type: tps
# Args: tps keyedlist containing the following keys:
#       MODE    run mode (”start”, “run” or “time”)
#       MSGID   message handle
#       ARGS    user-supplied arguments:
#               TIME : Numeric Number of Seconds to delay
#
# Returns: tps disposition list:
#          
#
###############################################
# Modification and Notes
###############################################
# 20100428 – Final cleanups for deployment to outreach site
#
# 20110805 – Terry Kellum – Added delay message for test.
#                            write_delay_msg_test
#                            read_delay_msg_test

proc write_delay_msg { args } {
   package require sqlite
   keylget args MODE mode               ;# Fetch mode
   keylget args CONTEXT context

   set dispList {} ;# Nothing to return
   
   
   ###################################################################
   # Main Code
   
   # Setup the db
   sqlite DelayDB ./delayed_transaction_database.db
   
   switch -exact — $mode {
       start {
           # Perform special init functions
   # N.B.: there may or may not be a MSGID key in args
   
   # Setup the Table
   DelayDB eval {CREATE TABLE IF NOT EXISTS dTransList(transnum integer primary key autoincrement,
insert_secs integer,
trans text,
meta text)}
   
   set NumOldRecs [DelayDB eval {select COUNT(*) from dTransList}]
   echo ”  >> tclproc: write_delay_msg -> Database Initialized. $NumOldRecs found.”
   echo ”                           Directory: [pwd]”
   echo ”  >> tclproc: write_delay_msg -> Context: $context n n”
   
       }

       run {
   # ‘run’ mode always has a MSGID; fetch and process it
           keylget args MSGID mh
   set msg [msgget $mh]
   set meta_fields [msgmetaget -rw $mh]
   set metalist “”
   foreach field $meta_fields {
lappend metalist [list $field [msgmetaget $mh $field]]
   }
   #echo $metalist
   set currsecs [clock seconds]
   if {[catch {DelayDB eval {INSERT INTO dTransList VALUES(NULL,$currsecs,$msg,$metalist)}} result] } {
echo ”  >> tclproc: write_delay_msg -> FAILED insert – Continuing message.”
lappend dispList “CONTINUE $mh”
   } else {
# If this is in UPOC Write, then we continue the message
if {[string equal $context pdupoc_write] == 1} {
   lappend dispList “CONTINUE $mh”
} else {
   lappend dispList “KILL $mh”    
}
   }
   
       }

       time {
           # Timer-based processing
   # N.B.: there may or may not be a MSGID key in args
       }
       
       shutdown {
   # Doing some clean-up work
   DelayDB eval {VACUUM}
}
   }

   return $dispList
}

proc read_delay_msg { args } {
   package require sqlite
   keylget args MODE mode               ;# Fetch mode
   keylget args CONTEXT context
   keylget args ARGS arguments
   keylget arguments TIME delay

   set dispList {} ;# Nothing to return
   
   
   ###################################################################
   # Main Code
   
   # Setup the db
   sqlite DelayDB ./delayed_transaction_database.db
   

   switch -exact — $mode {
       start {
           # Perform special init functions
   # N.B.: there may or may not be a MSGID key in args
   
   # Setup the Table
   DelayDB eval {CREATE TABLE IF NOT EXISTS dTransList(transnum integer primary key autoincrement,
insert_secs integer,
trans text,
meta text)}
   
   set NumOldRecs [DelayDB eval {select COUNT(*) from dTransList}]
   echo ”  >> tclproc: read_delay_msg -> Database Initialized. $NumOldRecs found.”
   echo ”                           Directory: [pwd]”
   echo ”  >> tclproc: read_delay_msg -> Context: $context”
   echo ”  >> tclproc: read_delay_msg -> Delay: $delay n n”
   
       }

       run {
   # ‘run’ mode always has a MSGID; fetch and process it
             #keylget args MSGID mh
   #Run mode not used here
       }

       time {
           # Timer-based processing
   # N.B.: there may or may not be a MSGID key in args
   # Set 60 Second delay
   set cutoff [expr [clock seconds] – 60]
   DelayDB eval {SELECT transnum, trans, meta FROM dTransList WHERE insert_secs < $cutoff} row {
#echo "num = $row(transnum)"
#echo "trans = $row(trans)"
#echo "meta = $row(meta)"
set mh [msgcreate]
msgset $mh $row(trans)
msgmetaset $mh PRIORITY 5120
DelayDB eval {delete from dTransList where transnum = $row(transnum)}
lappend dispList "CONTINUE $mh"
   }
   
       }
       
       shutdown {
   # Doing some clean-up work
   # vacuum done by creating table in write.
}
   }

   return $dispList
}

Looks like I wasn’t doing much with the metadata that I saved.  Perhaps that is because I did my xlate on the inbound side, and raw-routed on the outbound side.  I used an interval of 5 on the inbound UPOC (read_delay_msg).

[Richard Hart]

I’ve previously throttled inbound messaging by delaying the ACK sent back from Cloverleaf using the thread ‘phold_obd’ feature.

I’ve not attempted what you are trying achieve, but it may be worh investigating.

Note that this will only work in the ACK message is of type data.

[Rob Lindsey]

You could build a “wait” loop into your ACK to slow down the inbound data.  I would not recommend the   sleep   command or the   after    command because both of them block the TCL interrupter.  

There is however a set of commands that is CPU intensive

set WaitCnt 0 ; while { $WaitCnt < 10000000 } { incr WaitCnt }

Rob

[Author]

Replies