Speedup ack creation hl7Raw_ack.tcl

[Tim Davis]

We have encountered a situation in which our ACK creation is severely impacting performance in our production environment.  We have a process that sends messages raw, so ACKs are the only activity of any significance.  Using the vendor-supplied version of hl7Raw_ack.tcl, we’re able to send about 5 ACKs per second.  This seems to be using up nearly 100% of one of the 8 processors on our box.  5 messages per second is inadequate for the throughput we require on the process.  Can anyone recommend any method for improving the speed at which ACKs are created?

Topic

[Jim Kosloskey]

Tim,

Is the Source system backing up (in other words has messages queued up)?

If not, then perhaps 5 messges per second is as fast as the source system can generate messages.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Keith McLeod]

There could be more to this than you see on the surface.  I have had situations where the vendor system was using a slight variation on what it expected in the way of encapsulation.  For only one of their 20 interfaces for which they could not seem to fix, they required 2 additional carriage returns at the end of the ACK message.  We sniffed the line and found that when the ack was received by them, they waited 5 seconds before sending the next message.  The ack from our side went out nearly instantly.  Their side said we send the message as soon as we receive the ACK…finger pointing occurs here….  After sniffing the network and getting a low level programmer we isolated the delay to the time the adapter received the message to when the application received the message.  As with our pdls they will wait for the complete message a specified period of time.  This system waited 5 seconds….Once we added the non standard encapsulation the messages flowed without resistance.

[Gary Atkinson]

I use this proc for site to site feeds without problem.  I am on 5.5.

[Jim Kosloskey]

Keith,

Good point and simply more confirmation that the issue is rarely with Cloverleaf.

Additionally this is testament to the need for vendors to actually know their systems better and provide communication of their requirements.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Charlie Bursell]

Make sure you have the latest version.  It is now released as part of the root procs.

I know this is good because I wrote it  😀

Seriously, you should put a sniffer on the line and see what is *REALLY* going on.

Something to try.  Get a file of about 100 messages and run them through hl7Raw_Ack via hcitpstest and time it.  hcitpstest is much slower than the engine since it has to source the proc when it runs so if it is fasr there it should be much faster in the engine.

[Todd Lundstedt]

You can also test your ACK code, and engine code speed this way…

On a test site, set up four threads (adjust ports as necessary):

in_thread_a, server, port 8000, process1

raw_route to

out_thread_b, client, localhost, port 8001, process1

in_thread_b, server, port 8001, process2

raw_route to

out_thread_a, client, localhost, port 8000, process2

Make sure you have your ACK code in place.  Resend a dozen or so messages in one of the outbound threads and watch them go round and round.  Take notes of your msgs/sec, processor utilization, and any other metrics you want.  I don’t think you could ever achieve these numbers in real interfaces, as you have taken out the majority of the network, and any other application delays, but it will tell you just how fast your ACK code can ack while running in an engine.

Stop it all and turn on save-messaging on all of those threads, and let it run again (but not for too long, or you’ll fill up your disk!).  Note the differences in msgs/sec when Cloverleaf has to save stuff to disk.  With the .idx files, you can look at the time deltas between batches of message sends and report just how fast the ACK code works.

I used this setup when I was testing cluster failovers prior to implementation of a new cluster to help verify we would not lose a message during a failover.  I tried it with and without ACK code in place.  Without the ACK code, I only ever lost one message per failover.

On the other hand… as mentioned by others here, a sniff of the packets and appropriate analysis will tell you lots, too.

[Author]

Replies