Auto-bouncing threads

Hi,

We also ran into some problems with a couple of our connecitons where they get ‘clogged up’ and stop sending messages.  The root cause could not be located so a ‘temporary but semi-permanent fix’ was put in place.

An alert was setup to watch the thread and to call an external script when triggered.

The script then uses the hcicmd to stop and start the thread.

We created a generic script to allow us to pass in the process and thread names into the script so it can be reused

$1 == Process Name

$2 == Thread Name

Mark,

It should be just a matter of ‘exec’ ing the appropriate hcicmd when the desired condition is met.  We have several alerts configured like this to bounce a connection when the protocol status goes to ‘opening’ for x number of seconds.  You should be able to specify an action of ‘exec’ and then utilize the hcicmd command:

exec hcicmd -p myprocess -c ‘mythread pstop’;hcicmd -p myprocess -c ‘mythread pstart’

Hope this helps.

Jim Cobane

PHNS, Inc.

Hello,

I have a problem similar to this, the process stays UP (dosent go to OPENING) but the transactions stop going out.  A netstat on the recieving system shows 2 espablished connections from QDX to the same port on the recieving system, but QDX only shows 1 connection.  It’s like the recieving side never got the message to drop the old connection, but it started up the new one.

Any ideas would be appreciated

Mark, have you been able to determine anything with regards to the situation you described with remote system showing two established connections, but QDX with only one?  We are experiencing the same thing when QDX is operating as the client in a client server relationship.  I am beginning to think this is a Cloverleaf bug.  We are running a Win2003 server, and the remote system is Win2000.

I have been trying to use the exec below in an alert, but it dosent seem to work.  I set up a duplicate alert to just notify and I get the messages on the screen when my que gets to the settings.  I just can’t get the thread to cycle.

exec hcicmd -p process -c ‘thread pstop’;hcicmd -p process -c ‘thread pstart’

Mark,

 Our situations sound eerily similar.  My engine is Windows 2003.  We have a system (Datex Ohmeda, which is now part of GE) that is sending HL7 vital sign information to a port on my engine.  It will work great for about 4 hours, then for no reason messages stop coming across.  All I have to do is stop/start the inbound thread and everything is great until the next time.  The inbound thread stays in the UP status and nothing looks out of the ordinary.  Did you find a solution for your problem?

Ross

Mark,

I have had the same problem with threads staying in an ‘UP’ status, but never receiving a reply from the receiving system.  Many times this results in a run away process that keeps spitting out ‘read returned error 78’.  

What we have done here is to monitor within the recover_38 procs, the number of times we send to a specific connection without a response (assuming use of ACK/NAK).  Once we reach a specified threshold, we touch a file with the following name:

siteName.processName.threadName.command

We use an external script that checks for these files and executes the appropriate command.

The threads either come back ‘UP’ and we start sending or they come up in an ‘OPENING’ state.

By coupling the timeout on the connection with a max number of resends,  we can control how often a thread can get bounced.

Hope this helps.

Steve

Hello,

I have a problem similar to this, the process stays UP (dosent go to OPENING) but the transactions stop going out.

I’ll bet your Inbound Replies Timeout is set to -1.

If so, try changing it to a positive number like 60 seconds.  After the first message is sent to the broken connection and the ack doesn’t come back within the timeout, it should auto-reconnect and resend.  (Assuming you have reconnect and resend turned on, too.)

Mark R,

re your 2 ports in the netstat problem , we encountered similar issues running through a VPN where the VPN middleware apparently collapsed the tunnel because a keepalive parameter was not properly set.

We also found that in some circumstances the “zombie” port would not go away until the process controlling the thread was hcienginestopped.  Simply doing an hcicmd “pstop” on the thread wouldn’t release the hung port.

Mark,

Lay these two command on different alerts.

1) Let the first alert be configured to ( lastr (last recieved at 30 mins or whatever)

exec hcicmd -p process -c ‘thread pstop’

This will stop the INTERFACE thread,.

2) then have another alert which then checks for pstatus for 2 mins or so …if down

then confgure it

hcicmd -p process -c ‘thread pstart’

For some reason the alerts behave when on different alert lines. We are on 5.3.  Single line commands as Jim mentions worked on 5.2 but not 5.3

thanks

I have been trying to use the exec below in an alert, but it dosent seem to work.