Weird Routing Issue Encountered

[David Teh]

Hi folks,

We encountered a really weird issue with our Cloverleaf 5.8.7 on Solaris 10.

An inbound thread connects to multiple outbound threads.

The inbound thread routes ADT and BAR messages to the respective outbound threads.

For no reason, we realised one of the outbound threads was getting only the BAR messages. Even when we resent the ADT messages from the inbound thread, the messages don’t get routed. There are no error or warning messages anywhere, nor in the recovery/error dumps.

Other outbound threads are still getting the ADT messages.

We only managed to side-step the issue by creating a new outbound thread to the receiving system and recreating the routes.

What could have gone wrong?

 

 

Topic

[Jim Kosloskey]

A Tcl proc somewhere in the route filtering messages?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[David Teh]

Hi Jim,

No tcl in the route.

Those threads have been working since 2005.

Here’s the weirder bit: that old thread started getting the ADT messages after 9 hours. Again, without any change in config whatsoever.

[Charlie Bursell]

Use enable_all EO so you can see what is going on in the thread and in Xlate

[Ab Lugtenburg]

Hi David,

I reconized this issue with win 2016 and conversion from cloverleaf 6.1.2 to 19.1
As charlie mentioned i found with enable_all EO an issue with the HL7 definition.
The inbound uses the hl7 def. if you put on the hl7 as trix id

 

[David Teh]

Charlie Bursell wrote:

Use enable_all EO so you can see what is going on in the thread and in Xlate

 

Hi Charlie,

That old thread started getting the ADT messages after 9 hours (last week).

Will remember to turn on EO the next time it happens…. Thanks!

[David Teh]

Ab Lugtenburg wrote:

Hi David, I reconized this issue with win 2016 and conversion from cloverleaf 6.1.2 to 19.1 As charlie mentioned i found with enable_all EO an issue with the HL7 definition. The inbound uses the hl7 def. if you put on the hl7 as trix id

 

The config has been in place since 2005. And the other Outbound threads are receiving the ADTs just fine. For the other Outbound threads, routing is also by HL7 trx id and there are XLTs involved too.

Weird….
<p style=”text-align: center;”></p>

[Jim Kosloskey]

How did you determine the receiving system was not getting ADT Messages?

Did you observe them not in the Outbound SMAT or did the receiving system indicate they were not receiving ADT Messages?

The reason I ask is some time ago I had a system receiving multiple Message Types (something I don’t like to do on one connection) and they reported they were not receiving one of the Message Types.

I looked in the Outbound SMAT and the messages they reported missing were there. Of course they claimed there was no issue on their end.

Eventually they indicated they indeed began receiving the Message Types again.

With deeper inspection on their side it turned out (not to my surprise) they had multiple Message Type Handlers post receipt of messages (they had no SMAT like ability on their end so they could not determine if the messages actually arrived) and the handler for the missing Message Types had been ‘turned off’.

Problem solved.

Just thinking history may have repeated itself.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[David Teh]

Hi Jim,

Yes, we checked, it wasn’t in the SMAT.

For the problematic period of time, only BAR messages were in the SMAT. We even tried to resend one ADT from the Inbound thread but it didn’t appear on the Outbound. No errors too.

Weirdest case in my 16 years handling Cloverleaf.

[Jim Kosloskey]

If it is possible to resend one of the messages now it would be interesting to see if it now passes through.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[David Teh]

 

On that day, we basically left the original OB thread as it is except to switch the protocol properties to /dev/null. Another thread was created to send the messages to the destination system.

As mentioned earlier, “that old thread started getting the ADT messages after 9 hours. Again, without any change in config whatsoever.”

[Jim Kosloskey]

What I was suggesting is now that ADT messages are flowing to the connection previously not flowing if one of the messages which did not flow could be resent and that still does not succeed then I would suspect something in the message and that would give me something to check.

If the message now also goes through than that indicates perhaps it is not a message specific issue.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[David Teh]

Hi Jim,

Thanks for that.

On the day of the issue, after we created the new OB thread, we resent all the ADT messages from the IB thread (target destination: new thread) for the affected thread.

As mentioned, IB thread receives both BAR and ADT messages. Other threads were receiving both types of messages. The problem thread only sent out BAR messages for some reason, with nothing pending anywhere.

• This reply was modified 5 years ago by David Teh.

[Author]

Replies