Outbound thread will not send message

[Gary Atkinson]

I had a process panic with the dreaded Db Vista -921 Recovery error.  I did an lmclear.  One of outbound threads in the process that panic with not send an ack outbound.  The process log shows this below.  What should I do next?

[icl :tcpi:ERR /0:  results_cmd:01/21/2010 08:27:27] write failed: Broken pipe

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Since there are some error had occured while attempted to send an Ack back to client

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] We try to process the command anyway but no further Ack will be send back to Client

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Received command: ‘results_xlate xrel_post’

[cmd :cmd :INFO/0:results_xlate:01/21/2010 08:27:27] Doing ‘xrel_post’ command with args ‘‘

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Inrecoverable socket error.  Closing connection.

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Receiving a command

[icl :tcpi:ERR /0:  results_cmd:01/21/2010 08:27:27] write failed: Broken pipe

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Since there are some error had occured while attempted to send an Ack back to client

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] We try to process the command anyway but no further Ack will be send back to Client

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Received command: ”results_xlate’

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Cmd null in ”results_xlate’

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Inrecoverable socket error.  Closing connection.

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Receiving a command

[icl :tcpi:ERR /0:  results_cmd:01/21/2010 08:27:27] write failed: Broken pipe

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Since there are some error had occured while attempted to send an Ack back to client

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] We try to process the command anyway but no further Ack will be send back to Client

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Received command: ‘vp_res_cerner pstart’

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:27] Doing ‘pstart’ command on ‘vp_res_cerner’

[prod:prod:INFO/0:vp_res_cerner:01/21/2010 08:27:27] Applying EO config: ”

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:28] Inrecoverable socket error.  Closing connection.

[pdl :PDL :ERR /0:viewpoint_res:01/21/2010 08:27:29] read failed: Connection reset by peer

[pdl :PDL :ERR /0:viewpoint_res:01/21/2010 08:27:29] read returned error 73 (Connection reset by peer)

[pdl :PDL :ERR /0:viewpoint_res:01/21/2010 08:27:29] PDL signaled exception: code 1, msg device error (remote side probably shut down)

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:30] Receiving a command

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:30] Receiving a command

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:30] Received command: ‘results_xlate xrel_post’

[cmd :cmd :INFO/0:results_xlate:01/21/2010 08:27:30] Doing ‘xrel_post’ command with args ‘‘

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:30] Receiving a command

[cmd :cmd :INFO/0:  results_cmd:01/21/2010 08:27:30] Command client went away.  Closing connection.

Engine idle — 01/21/2010 08:27:44

Topic

[Gary Atkinson]

Ok I did a complete site clean-up and cleared and started back up the databases.  I still keep getting the same errors on the one outbound thread.

[Jim Kosloskey]

Looks to me like the receiving system has an issue.

Has you thread really started (come to an ‘Up’ state)?

Because these lines indicate the receiving system has shut down the connection:

[pdl :PDL :ERR /0:viewpoint_res:01/21/2010 08:27:29] read failed: Connection reset by peer

[pdl :PDL :ERR /0:viewpoint_res:01/21/2010 08:27:29] read returned error 73 (Connection reset by peer)

[pdl :PDL :ERR /0:viewpoint_res:01/21/2010 08:27:29] PDL signaled exception: code 1, msg device error (remote side probably shut down)

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Gary Atkinson]

Yes after some time the thread did go to UP status.  After investigating on the receiving side they are having an issue.  Any ideas on why this would cause the process to panic?

[Jim Kosloskey]

Normally I would not think this should cause the process to panic.

It may be something unusual in the configuration that you have set up or it could be unrelated.

When the panic happened does the log lead you to believe this issue was related?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Gary Atkinson]

This was the only thread in the process that was having the issue.  I have contact the receiving system vendor.  I have never had this kind of issue before, so I am stumped  😳

[Sundeep Kumar]

We have seen an occasional process panic when the receiving vendor is troubleshooting/working on his listening ports and gets his listner into a debugging mode. Seems like erratic responses back from the receiving thread can send the cloverleaf process eventually to panic.

[Gary Atkinson]

Thanks for information.  I am now in a pointing finger fight with the vendor.  “They” are saying CL is dropping the connection.  I turned up PDL logging on the thread and sent this to CL support.  I am at the end of my level of knowledge  🙄

[Sundeep Kumar]

Is this connection over VPN ? Then it could be a whole different host of problems. We have had problems with VPN connections and I have heard similar opinion on this forum. What seems to happen is that the underlying connection seems to dissapper while both the sender and the reciver think they have active connections at the protocol level.

[Gary Atkinson]

Not over a VPN.  BUT I know what you mean…I hate VPN’s 😈

[Charlie Bursell]

You will always have this type of finger pointing without a sniffer.  Put a sniffer on the line so you can see what is on the wire.  No arguing with that either way

[Robert Milfajt]

AIX has a built in sniffer utility called iptrace.  The man page on it is all I needed to get it going.

Robert Milfajt
Northwestern Medicine
Chicago, IL

[Michael Hertel]

The error messages regarding sending acks in the log is not about HL7 tcp acks across an interface.

It is about the magic internals of the engine communicating with itself/hostserver/client gui.

Example: you click shutdown process

If you just started the process and it is churning thru thousands of messages in the recovery database, it will take a while to stop churning and process the stop request.

There is a timeout involved for the engine to receive back a response for a running process. If it doesn’t get that response within that time, you get these broken pipe “error had occured while attempted to send an Ack ” messages, yet the command will eventually be processed and the process stops.

Perhaps someone could explain this better than I just did.

[Author]

Replies