Intersite message loss

[Jason Russell]

Wanted to see if you guys had any ideas before moving on to support for an ‘odd’ issue we’re having. We’re migrating to cloverleaf from eGate and we’re doing it kind of blindly. Our team has taken all of the classes and all Level 3 and TCL certified. Some of our legacy ideals are probably hampering our move forward but that’s what the overall management/leads have decided for us going forward. A few notes to start us off:

• CL version 20.1.0.0 (We’re planning on upgrading when we finally move to our formal test systems)
• We do not use SMAT currently.
• Intersite communications are all TCP/IP. I’ve actually tested with the intersite communication set up, not sure if switching to that would be any better/faster. There was a reason we chose TCP/IP, but I’m not recalling it at the moment.

The basic idea of what is happening is we are ‘losing’ messages when we restart certain inbounds from intersite communications. SiteA sends ADT to SiteB before being processed and sent to the vendor. That A to B connection is where the oddities are happening. The way we know it’s being lost there is we have a kind of ‘custom’ journaling set up in place of SMAT (Legacy item, our policy is to hold messages forever basically). We journal in two main places. The Inbound (in TPS Inbound Data, after the ack script) and outbound (in Prewrite Procs on the outbound thread).

What happens is we apply an update to a thread in SiteB, restart the thread, and we will sometimes lose a message. It all seems to come down to timing. An overview of our thread setup:

SiteA

ADT_in –> ADT2siteb_out (ADT_in.journal and adt2siteb_out.journal are created)

SiteB

ADTfromSiteA_in –> ADT2vendor_out (ADTfromSiteA_in.journal and ADT2vendor_out.journal are created).

When the restart happens, the message will flow and be written in both of SiteA’s journal files, but will NOT write to siteB’s journal files. As far as we can tell, it looks like the sending thread (2siteB) sends the message, the receiving thread (fromSiteA) receives the message, sends the ACK but does not journal the message, put it in the Recovery database in siteB, nor does it log any errors in either site. It looks like the shutdown causes it to vanish during those HL7 ACK handshakes.

Has anyone seen anything like this? I know the explanation is probably a bit wonky, but I will clarify the best I can. If the community does not have any ideas, support is our next stop.

Topic

[Peter Heggie]

When you say “intersite” I can’t help but think of the capability that Cloverleaf has to send a message from a “source” thread on site A to a “destination” thread on site B.  But I’m thinking that is not the case here. Here you have a source and a destination thread on site A and a source and destination thread on site B. This is common. Especially using TCP/IP. So I suspect that some aspect of the configuration is not quite right.

Please check the following:

• destination thread on site A is defined as the Client
• that you are specifying PDL tcp/ip and the PDL on the site A destination thread is the same as the PDL on the site B source thread.
• on the Outbound tab of the site A destination thread has Inbound Replies set to Await Replies and you have a timeout value set (45 seconds?), and a TCL proc specified for the TPS Inbound Reply (like “hcitpsmsgkill”).
• on the Inbound tab of the site B source thread, in the Inbound Data window, you have an ACK proc set in the TPS Inbound Data window (ex: “raw_ack”).
• source thread on site B is defined as the Server
• make sure Auto Reconnect is enabled

Peter

Peter Heggie

[Jim Kosloskey]

Have you turned the engine output all the way up and validate the journalling proc is being executed (or not) as well as tracing the disposition of the IB message?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jason Russell]

We are not using the “intersite” communications as you described, as of now our usage of “intersite” simply means site to site communications (not to another server/vendor). We are not currently using the PDL protocol, but standard TCP/IP.

Going over your settings, it looks like we have the ‘await replies’ set on the INBOUND thread, but not the OUTBOUND threads, which is probably the issue. I cannot believe that was set, and it looks like we’ve been setting it incorrectly the entire time. I’m not sure WHY this was never set correctly unless someone misheard what to set in training. That would actually explain why it only happens on reboots. Message is sent before the interface realizes the other side is down, but no acknowledgement is actually generated. Otherwise it works fine as there is no actual interruption in the service.

@Jim we may have turned it up, but potentially not all the way up. I’ll have to look at it again. Right now it’s not an issue since we’re not fully in production (yet), and only happens irregularly when we restart a thread, and has to happen at a specific time in the communication process, from what we’ve seen. Some shutdowns are fine. However, I think what Peter pointed out could very well be our culprit.

[Jim Kosloskey]

Jason,

I think you are correct in that the ‘Await replies’ needs to be set on the OB thread, ‘Outbound’ Tab, but also you need to check ‘Outbound Only’ check box on the ‘Inbound’ Tab of the OB thread.

Most likely incorrectly set as a result of a misunderstanding.

Jim

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies