remote side probably shut down in CL 2022.09

[Colin]

Hello All,

We have upgraded to a higher cloverleaf version  a couple of months ago to version 2022.09

Since then we have problems with our ADT threads. In the engine log of there is an error

[pdl :PDL :DBUG/0:ITransport_ADT:11/07/2023 11:14:55] Evaling:
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc hci_pd.write { info } {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] global MsgId
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylget info message MsgId
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset continuations ok write.done
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset continuations error write.error
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset continuations timeout

[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_send basic-msg

]] $continuations
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc write.done {info} {}
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc write.error {info} {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_report_exception 1 “write failure”
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_set_result_code 1
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc write.timeout {info} {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] global MsgId
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] msgmetaset $MsgId FLAGS {{proto_timeout 1}}
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_set_result_code 1
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc hci_pd.read {info} {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset continuations basic-msg read.done
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset continuations error read.error
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset continuations timeout

[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_receive $continuations
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc read.done {info} {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset accept text

]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylset accept end [keylget info end]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_accept $accept
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc read.error {info} {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] keylget info type type
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] switch -exact — $type {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] input-error {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_report_exception 3 “device error (remote side probably shut down)”
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_ignore_input -all
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] no-match {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_ignore_input 1
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_ignore_input -until \xb
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] default {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_report_exception 2 “unknown fail: $type”
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_ignore_input -all
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–]
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] proc read.timeout {info} {
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_ignore_input 1
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] hci_pd_ignore_input -until \xb
[pdl :PDL :DBUG/0:ITransport_ADT:–/–/—- –:–:–] }

 

On our production platform all is working normal, this on happens in out new test platform.

We don’t what causes the error all threads are up and working normal

Please assist and thanks

Greeting,

Colin

Topic

[Peter Heggie]

I’m not familiar with this level of debugging with tcp/ip pdl –

• what platform / os level are you on
• is this inbound or outbound; client or server or multi-server
• which PDL are you using
• are you using IPv6/v6, SSL, Close After Write, Driverctl
• are you using Auto re-connect, and/or Delay Connection Until Needed
• do you know what the Keep Alive is on your side, and on the other side?
• is the other side inside your network?

Also, and I know this is probably not a factor, but what trxid type do you use (HL7?), do you have an ACK proc, and what is the OS on the other side?

At first glance it looks like expected packets are not getting received, but I’m not a network expert.

So are you running the same Cloverleaf version in Test and Prod? Is Test on a different OS or OS version? Is it in a different network? This might not be an issue with Cloverleaf.

Peter Heggie

[Tim Pancost]

Hey, Colin,

Doesn’t look to me like you actually have any error.  Looks like you’re in debug mode, so it’s listing out the entire PDL code.  That “error” you’ve highlighted is what would happen if the switch statement evaluates to “input-error”.  Other options are “no-match” and “default”.

Seems like you have your engine output jacked WAY up on your test box.

HTH,

TIM

Tim Pancost
Trinity Health

[Peter Heggie]

We also are now receiving new “errors” from PDL processing after we upgraded to 2022.09; everything seems to be working so we ignore them.

ex:

[xlt :xlat:WARN/0: c62_xlate:11/07/2023 07:14:57] [0.0.76442748] Delivery to destination thread to_rhio_direct failed. Requeuing msg. iclErr=14
[icl :icl :ERR /0: c62_xlate:11/07/2023 07:14:57] Transport to td ‘to_rhio_direct’ busy

Peter Heggie

[Jeff Anderson]

After upgrading to 2022.09 we’re getting the same thing:

[0.0.175920684] Delivery to destination thread patient_matching failed. Requeuing msg. iclErr=14

Is this a new ‘normal’? As far as we can tell everything seems to be working.

[Charlie Bursell]

Have you notified Support?

[Jeff Anderson]

Just got a response. We are currently on 2022.09.01 and was told to upgrade to latest version in which this “issue” is fixed. So we’re good now.

[Author]

Replies