Disappearing messages

[Mark Brown]

I can’t figure out what might be going on here. I got a call about an account missing from an ancillary system.  I checked SMAT for that thread and no messages went out for account.  The hospital system sent the ADT, but this ancillary system (and I’ve found others on different threads also) didn’t get the messages.  There are no tcl procs or xlates that might block or otherwise filter the messages for these particular ancillary systems.

I was able to pull the messages out of SMAT for the thread receiving the messages from the hospital system and then send them directly to the ancillary system and they went ok.

I don’t even know where to start to find out what’s happening.

We’re running a virtual Windows Server 2008 with CL6.02

Topic

[Jim Kosloskey]

Check to see if another thread (if there are multiple receiving threads) got more than one of this message.

So inbound-a routed to outbound_1 and outbound_2; outbound_1 has the problem, check outbound_2 to see if it got 2.

Above is a simplification so you may have a more complicated configuration – thus check all of the outbounds.

Also check the Recovery DB for the problem thread to make sure the message is not stuck there.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jeff Dinsmore]

If you’re doing processing on your inbound thread, perhaps there was a failure in a tcl proc that prevented delivery to one or more systems.

For example, let’s say you have your inbound A routing to destination systems B, C and D.

Let’s also assume in the A thread you have raw routes to B, C and D – in that order – with some number of TCL procs called for each of them.

If there is an unhandled error in one of the TCL procs for the route to B, the current message will not be routed to C or D.

That’s the reason I never do any processing on an inbound thread.  I only use them for raw distribution to outbound pool threads that feed a single system.  That way, if there is some unhandled error, it only affects that one system.

So, check your process error log around the time the message was not delivered to see if you have any errors.

NOTE:  This is how Cloverleaf behaves in our 5.6 environment – I’m extrapolating/assuming it’s the same in 6.x

Jeff Dinsmore
Chesapeake Regional Healthcare

[Mark Brown]

Here is something I’ve discovered and it totally makes no sense to  me. Hopefully there’s a solution.

I looked in the error database. It was huge.  I wrote all the messages to a file using -L -c to get details.  This is what is strange. The message I was looking for was in the error database. The destination thread listed is not the thread the message is missing from. The error message with the error is from a tcl proc called from an xlate a different thread in the same process.

So it seems all the messages with patients without a bed assignment are failing for all threads even though the xlate is on only one thread. This thread has been in use at least since 2004 (last edit date) and probably long before that and has been running without this problem.

If I replace this xlate, will this solve the problem? Or just edit it and save it so it gets over-written?

The messages have been going into the error database on this process since putting 6.02 in production at our site.

Any ideas for best option?

[Mark Brown]

Here is the error message. The line is generated by the xlate.

Patient was not assigned to a bed. Cancelling Translation!

Previous recovery db state is 5.

[James Cobane]

When a message fails in translation, it will fail for all destinations within that same process.  So, even if it is not your xlate/thread that is having the issue, it will not get the message if there is an error in one of the translations in the same process.  This could be a good thing or a bad thing, depending on your perspective, but it is how the engine was designed/functions.

You’ll need to correct/remediate the xlate that is failing the message.

Jim Cobane

Henry Ford Health

[Mark Brown]

I took out the call to the tcl proc that checks for a valid room from the xlate and used a pre-proc that does the same thing to kill the message before it reaches the xlate if there’s not a valid room.  Now messages without a valid room are going to thread that had the missing messages (it doesn’t care about rooms).

If the system is designed to function this way, why did it only manifest itself after the upgrade and not in the previous 10+ years?

[Jim Kosloskey]

Mark,

Is the proc issuing an xpmerror call and is the Xlate Action error disposition of the invoking Xlate Action set to anything besides ‘skip’.

If so, then prior to 6.0 setting an xpmerror and an error disposition in an Xlate Action to exploit the xpmerror has not worked and was corrected in 6.0.0 I think.

So the proc was probably always doing what it does but the Xlate was not honoring what the proc does – now the Xlate does.

I believe there was mention of this in the Release Notes – if you jumped from pre 6.0 to 6.0.2, then you should have read all of the release notes and then (if there is a mention in the release and I am not hallucinating) taken whatever action needed as part of the upgrade.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Mark Brown]

it is calling xpmerror, but the disposition is set to skip

[Jim Kosloskey]

OK well skip is supposed to mean skip the Action and continue with the Xlate, error is supposed to mean stop the translation and send the message to the error DB.

So did you actually get a Tcl error as well as the display from the xltp proc or just the error from the xltp proc?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Mark Brown]

The msgstate = Tcl callout abort (303)

Then the message from the tcl proc: Patient was not assigned to a bed. Cancelling Translation.

And then

Previous recovery db state is 5.

I had another thread in that process with xlates starting to do the same thing. I moved it to it’s own process for now and nothing is being kicked to the error database at the moment.

The destination thread on all the error messages is a thread that currently has all messages to it being killed.

[Jim Kosloskey]

On the xpmerror command what is the severity being set to?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Mark Brown]

I don’t see a severity being set anywhere.

[Jim Kosloskey]

It would be right on the xpmerror command – can you share the xpmerror invocation in the Tcl proc?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Mark Brown]

Here it is.  Is “curdetail” the severity?

xpmerror $xlateId curdetail “Patient was not assigned to a bed. Cancelling Translation!”

[Jim Kosloskey]

Yes and that means send the message to the error db which is what is happening.

If you would like to control this from the Xlate Action, then change curdetail to action and in the Error: area of the Xlate Action, select which option you want.

See this is now working the way it should have – but not the way you were used to.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies