Best tclproc error handling method

[Joshua Cooper]

Hi all,

Relatively new to Cloverleaf.  We use 6.0.2. Many of our tclprocs are embedded in inbound or outbound tps or Xlates (as intended) and we are finding that when there are errors with them, it goes largely unnoticed. Typically we are notified of a problem and find the errors hidden in the process logs. I’m looking for help on the ‘best’ strategy – ex. perhaps the thread queues up messages when a proc errors so that Alerts can be triggered and notifications & actions taken to correct immediately.

Here’s a scenario:

inbound ORU from third party going into our EHR.  third party device simultaneously places a PDF file with acct number in shared dir.  In Xlate, we call a proc to search that dir for the respective PDF file, then copy and rename to link them.  Every so often we see errors where someone had the PDF open somewhere else and the proc fails because it can’t modify the open file.

Another scenario:

tpsproc uses odbc connection to query data to add to strip down message and then rebuild msg and send on. odbc connection fails for whatever reason.  not as common but you get it.

It’s been suggested to us that we rethink how we’ve built our entire engine’s infrastructure.  maybe we take out all the procs and built threads in between inbound and outbound and code them in such a way that if the proc fails then the thread goes into error state and messages queue up. I’m aware this method of ‘raw inbound – data manip – outbound’ is tried and true but cloverleaf provides all of these clever ways for us to embed procs and all of this was already built when I inherited the system.

 

Thoughts?

Email: joshua.cooper@allscripts.com
-------------------------------------
Allscripts
Mobile, AL

Topic

[Charlie Bursell]

The biggest problem I have seen with Tcl procs is that they do not address anomalies.  People write procs as if they will never fail.  You should play devil’s advocate and be able to catch any error, especially file  or DB failures.  Remember, as Murphy said, anything that can happen usually will happen.

Once you catch the errors, how you handle them is up to you.  You could write a message to the log file.  If you do put a couple of linefeeds before and after the message so it stands out.  You could send an email.  You could shut down the thread.  If you do that I would shut down the process since the process may be busy and not shut down the thread.  You can do a combination of these.  There is no one size fits all.

What you should never do is to allow a failure to come down around your ears.  Allow it to fail on your terms.  It can be handled.

[Joshua Cooper]

Charlie Bursell wrote:

The biggest problem I have seen with Tcl procs is that they do not address anomalies. People write procs as if they will never fail. You should play devil’s advocate and be able to catch any error, especially file or DB failures. Remember, as Murphy said, anything that can happen usually will happen. Once you catch the errors, how you handle them is up to you. You could write a message to the log file. If you do put a couple of linefeeds before and after the message so it stands out. You could send an email. You could shut down the thread. If you do that I would shut down the process since the process may be busy and not shut down the thread. You can do a combination of these. There is no one size fits all. What you should never do is to allow a failure to come down around your ears. Allow it to fail on your terms. It can be handled.

Totally agree. The preferred outcome is that if a proc fails for one reason or another, the message does not get sent and thread doesn’t stay up.  We want it to change thread status to error/down, and queue the message back into it’s original state just before the error (1-11).  This way, we’re able to see the error in the process log and perhaps write some TCL in the alert or proc to perform environment checking and proceed if all good.

Email: joshua.cooper@allscripts.com
-------------------------------------
Allscripts
Mobile, AL

[Charlie Bursell]

Write your specific requirements in detail and then implement them.

The message will not stay in the current state if it is errored.  If the message is errored it will go to the Error DB and be removed from the queue (recover database) all together.  You could later resend from the Error DB.  It would be best if you allowed it error on your terms then there are many options.   All doable with proper coding

[Jeff Dinsmore]

Like Charlie says, catch/handle errors rather than letting them explode.

Relative to your failed ODBC problem:

A while back, we had several live interfaces that were reading realtime from our EMR’s DB to augment messages.

Since Cloverleaf doesn’t have a wait-and-retry message disposition, I built a FIFO queue with Tcl and an SQLite database to do that.

Every message was first written to the SQLite DB.

Then, the process reads any messages in the DB and processes them in order.

If the connection to the EMR DB is available, the messages are processed with the DB augmentation.  If the DB is not available, it kills the message and bails.

During normal operations, one message is written to SQLite and it’s immediately re-read and processed.

During an EMR DB outage – such as during an upgrade downtime – messages queue up in SQLite until the EMR DB is available again.

If inbound message volume is low, you’ll want to have a timed process to send queued messages so it doesn’t have to wait for an inbound message to trigger that.

You could likely do something similar to fix your unavailable PDF problem.

Jeff Dinsmore
Chesapeake Regional Healthcare

[Jim Kosloskey]

I have used email as a mechanism to communicate detected problems to carbon-based entities.

Of course it depends on whether your integrations are centrally monitored or not as well as the mechanism for centrally monitoring.

If using Global Monitor, for example, there may be a built in mechanism for communicating with the monitoring identities and that method would be deployed – perhaps along with email.

I believe the email should include the site name (if multiple site architecture for sure), Process name, thread name, Xlate name (if in an Xlate), an eye catcher so that the information in the log can be easily found, and a reasonably descriptive message indicating the condition and the actions taken.

It is also useful I think to place the eye catcher and other meaningful information in the USERDATA Metadata field prior to ERRORing the message (if the message is to be errored). That way the appropriate message can be more assuredly found in the Error DB for resend.

The eye catcher can include the date/time (YYMMDDHHMMSS) and a counter (Counter file) something like this: YYMMDDHHMMSS_nnnnn or anything that makes the error unique and easier to match up to the email.

Be aware inside an Xlate the proc cannot directly ERROR a message, rather the disposition can be set and then the Xlate Action wherein the proc is invoked sets the action to be taken if an error is reported. Also I think this functionality did not work in many Cloverleaf releases but has been repaired in later releases so you would want to test that.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies