Auto-bouncing threads

[Mark Brown]

We’re running Cloverleaf 5.3.1 on a Windows 2003 server.  We’re currently testing our connection to a remote site (Canopy) by sending a feed from our production server to their test system (through our test server.)  For whatever reason, probably reduced traffic since they’re only wanting two patient types, the connection gets dropped.  A simple bounce starts it up.  I tried to set up an alert to bounce the thread if it goes down, but it doesn’t do anything.

What is the trick to get an alert to bounce a thread?  

Thanks…

Topic

[David Dillard]

Hi,

We also ran into some problems with a couple of our connecitons where they get ‘clogged up’ and stop sending messages.  The root cause could not be located so a ‘temporary but semi-permanent fix’ was put in place.

An alert was setup to watch the thread and to call an external script when triggered.

Example:

{VALUE opque} {SOURCE Test1} {MODE actual} {WITH -1} {COMP {> 10}} {FOR {nmin 30}} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {bumpThread.sh Testoutbound Test1}}}}

The script then uses the hcicmd to stop and start the thread.

We created a generic script to allow us to pass in the process and thread names into the script so it can be reused

$1 == Process Name

$2 == Thread Name

#; stop the thread

hcicmd -p $1 -c “$2 pstop”

#; Set for specific timespan but could be changed

#;     to watch and wait for the thread to go down before

#;     bringing it back up again

sleep 30

#; start the thread

hcicmd -p $1 -c “$2 pstart”

[James Cobane]

Mark,

It should be just a matter of ‘exec’ ing the appropriate hcicmd when the desired condition is met.  We have several alerts configured like this to bounce a connection when the protocol status goes to ‘opening’ for x number of seconds.  You should be able to specify an action of ‘exec’ and then utilize the hcicmd command:

exec hcicmd -p myprocess -c ‘mythread pstop’;hcicmd -p myprocess -c ‘mythread pstart’

Hope this helps.

Jim Cobane

PHNS, Inc.

[Mark Rilling]

Hello,

I have a problem similar to this, the process stays UP (dosent go to OPENING) but the transactions stop going out.  A netstat on the recieving system shows 2 espablished connections from QDX to the same port on the recieving system, but QDX only shows 1 connection.  It’s like the recieving side never got the message to drop the old connection, but it started up the new one.

Any ideas would be appreciated

[Mark Perschbacher]

Mark, have you been able to determine anything with regards to the situation you described with remote system showing two established connections, but QDX with only one?  We are experiencing the same thing when QDX is operating as the client in a client server relationship.  I am beginning to think this is a Cloverleaf bug.  We are running a Win2003 server, and the remote system is Win2000.

[Mark Rilling]

I have been trying to use the exec below in an alert, but it dosent seem to work.  I set up a duplicate alert to just notify and I get the messages on the screen when my que gets to the settings.  I just can’t get the thread to cycle.

exec hcicmd -p process -c ‘thread pstop’;hcicmd -p process -c ‘thread pstart’

[Ross Stolle]

Mark,

 Our situations sound eerily similar.  My engine is Windows 2003.  We have a system (Datex Ohmeda, which is now part of GE) that is sending HL7 vital sign information to a port on my engine.  It will work great for about 4 hours, then for no reason messages stop coming across.  All I have to do is stop/start the inbound thread and everything is great until the next time.  The inbound thread stays in the UP status and nothing looks out of the ordinary.  Did you find a solution for your problem?

Ross

[Mark Brown]

Yeah, I finally got a solution.  I used the suggestions I got here, and studied how we were calling hci commands through scripts when an alert fired (my predecessor set up a slick system of VB scripts that monitor and manage the engine, then he left.)

I set up an alert that if the queue starts to back up (I gave it a time of 5 minutes) it runs a series of scripts that ulimately calls a wscript shell to run the hci commands ‘ hcicmd -v -p -c “ pstop” ‘ followed by a short pause then it runs ‘ hcicmd -p  -c “ pstart” ‘

The scripts are kinda hard to follow, but they basically set up the environment variables then builds a command line to be executed through wscript.  I’m not sure that helps…

[Steve Carter]

Mark,

I have had the same problem with threads staying in an ‘UP’ status, but never receiving a reply from the receiving system.  Many times this results in a run away process that keeps spitting out ‘read returned error 78’.  

What we have done here is to monitor within the recover_38 procs, the number of times we send to a specific connection without a response (assuming use of ACK/NAK).  Once we reach a specified threshold, we touch a file with the following name:

siteName.processName.threadName.command

We use an external script that checks for these files and executes the appropriate command.

The threads either come back ‘UP’ and we start sending or they come up in an ‘OPENING’ state.

By coupling the timeout on the connection with a max number of resends,  we can control how often a thread can get bounced.

Hope this helps.

Steve

Mark Rilling wrote:

Hello,

I have a problem similar to this, the process stays UP (dosent go to OPENING) but the transactions stop going out.

[Nathan Martin]

I’ll bet your Inbound Replies Timeout is set to -1.

If so, try changing it to a positive number like 60 seconds.  After the first message is sent to the broken connection and the ack doesn’t come back within the timeout, it should auto-reconnect and resend.  (Assuming you have reconnect and resend turned on, too.)

[Ed Mastascusa]

Mark R,

re your 2 ports in the netstat problem , we encountered similar issues running through a VPN where the VPN middleware apparently collapsed the tunnel because a keepalive parameter was not properly set.

We also found that in some circumstances the “zombie” port would not go away until the process controlling the thread was hcienginestopped.  Simply doing an hcicmd “pstop” on the thread wouldn’t release the hung port.

[sanjay suri]

Mark,

Lay these two command on different alerts.

1) Let the first alert be configured to ( lastr (last recieved at 30 mins or whatever)

exec hcicmd -p process -c ‘thread pstop’

This will stop the INTERFACE thread,.

2) then have another alert which then checks for pstatus for 2 mins or so …if down

then confgure it

hcicmd -p process -c ‘thread pstart’

For some reason the alerts behave when on different alert lines. We are on 5.3.  Single line commands as Jim mentions worked on 5.2 but not 5.3

thanks

Mark Rilling wrote:

I have been trying to use the exec below in an alert, but it dosent seem to work.

[Author]

Replies