Connection hanging after daily vendor restart.

[Bonnie Giesler]

At midnight, our radiology system does a restart on all the connection jobs.  If Cloverleaf is in the process of sending an ADT/ORM in the middle of this restart (of which takes like 1 second), RIS tries to connect but Cloverleaf keeps resending until the alert is triggered.

I have auto-connect checked and cloverleaf set for unlimited retries but it does not seem to understand that the connection needs to be reconnected. And in fact the logs indicates that.

This is the log from one of the many night that I have babysat this interface.  If there are no messages getting sent at the time that the RIS restarts, there are not issues and connections are re-established.

10 01:00:47] PDL signaled exception: code 1, msg device error (remote side probably shut down)

[pdl :PDL :ERR /0: ORU_Meditech:11/13/2010 01:02:11] read returned error 0 (Success)

[pdl :PDL :ERR /0: ORU_Meditech:11/13/2010 01:02:11] PDL signaled exception: code 1, msg device error (remote side probably shut down)

Engine idle — 11/13/2010 01:02:32

Engine idle — 11/13/2010 01:02:42

Engine idle — 11/13/2010 01:02:53

Engine idle — 11/13/2010 01:03:03

Engine idle — 11/13/2010 01:03:13

Engine idle — 11/13/2010 01:03:24

Engine idle — 11/13/2010 01:03:34

Engine idle — 11/13/2010 01:03:44

Engine idle — 11/13/2010 01:03:55

Engine idle — 11/13/2010 01:04:05

Engine idle — 11/13/2010 01:04:15

Engine idle — 11/13/2010 01:04:26

Engine idle — 11/13/2010 01:04:36

Engine idle — 11/13/2010 01:04:46

Engine idle — 11/13/2010 01:04:56

Engine idle — 11/13/2010 01:05:07

Engine idle — 11/13/2010 01:05:17

Engine idle — 11/13/2010 01:07:27

At this time the icon is red but still opening so I just did a start

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:24] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:24] Received command: ‘ORM_ADT_Syngo pstart’

[cmd :cmd :WARN/0:rad_decorah_cmd:11/13/2010 01:09:24] Thread is already running ‘ORM_ADT_Syngo’

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:24] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:24] Command client went away.  Closing connection.

Engine idle — 11/13/2010 01:09:34

I then did a stop and start and finally that worked

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:47] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:47] Received command: ‘ORM_ADT_Syngo pstop’

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:47] Doing ‘pstop’ command on thread ‘ORM_ADT_Syngo’

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:47] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:47] Command client went away.  Closing connection.

[prod:prod:INFO/0:ORM_ADT_Syngo:11/13/2010 01:09:47] Checking for leaked handles in the General interpreter…

[prod:prod:INFO/0:ORM_ADT_Syngo:11/13/2010 01:09:47] Checking for leaked handles in the TPS interpreter…

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:51] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:51] Received command: ‘ORM_ADT_Syngo pstart’

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:51] Doing ‘pstart’ command on ‘ORM_ADT_Syngo’

[prod:prod:INFO/0:ORM_ADT_Syngo:11/13/2010 01:09:52] Applying EO config: ”

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Command client went away.  Closing connection.

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Received command: ‘rad_decorah_xlate xrel_post’

[cmd :cmd :INFO/0:rad_decorah_xlate:11/13/2010 01:09:52] Doing ‘xrel_post’ command with args ‘‘

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Command client went away.  Closing connection.

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Received command: ”rad_decorah_xlate’

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Cmd null in ”rad_decorah_xlate’

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Receiving a command

[cmd :cmd :INFO/0:rad_decorah_cmd:11/13/2010 01:09:52] Command client went away.  Closing connection.

Topic

[Troy Morton]

Hi Bonnie,

I have seen this before.

[Russ Ross]

There is also the ablitly to wirte a TCL proc to proactively do whatever you want when a message times out and gets resent.

For example, you could write each resent message out to a file or simply increment a counter if that is all you need.

In our case, we use that resend counter to trigger an alert and recycle the thread.

This has worked very well for us at night when only one message might be queued up.

I’ve had several other hospitals contact me about this and implement this solution with great satisfaction, too.

I’ve also suggested cloverleaf be enhanced to have a built in alert based on number of resends and expect someday it will be enhanced to be included.

As you know once you have to start resending the same message more than once you are headed for trouble so why night cycle the thread right then to take care of the problem ASAP.

Here is one of my old posts illustrating how a resend TCL proc sends out an alert once a message is resent x number of times, which you could use as a tangible example and leverage for you own needs:

   <a href="https://usspvlclovertch2.infor.com/viewtopic.php?t=1862&#8243; class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?t=1862

You might also want to read this URL since resend procs can only be used if opting not to use the built in resend capability with later versions of cloverleaf from what I currently understand:

    <a href="https://usspvlclovertch2.infor.com/viewtopic.php?t=4212&highlight=tspcheckresendcount&#8221; class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?t=4212&highlight=tspcheckresendcount

Russ Ross
RussRoss318@gmail.com

[Author]

Replies