Delivery to xlate thread failed. Requeuing msg.

[Kevin Scantlan]

We are on version Cloverleaf v 6.1.2 with AIX 6.1 .

Topic

[Rob Abbott]

So a protocol thread is unable to deliver to the xlate thread within the same process?  That is strange.

Is the process name unusually long or have special characters?

Are you able to contact the xlate thread when this is going on (e.g. hcicmd -p -c “_xlate purgex”)

I would suggest you open a case with Support through Xtreme to get this resolved and determine if anyone else has seen this (I haven’t).  

We would certainly hate to see you back out an upgrade!

Rob Abbott
Cloverleaf Emeritus

[Kevin Scantlan]

We have a case open.  It’s my guess that for some reason, version 6.1.2 is not handling a abort and it’s corrupting the ICL thread.  When we bring the process back up is when we start getting the “Delivery” messages.  When we clean up the ICL, it goes away.

[Kevin Scantlan]

Rob,

I did not try to go a hcicmd command, but I presume that when we brought the process back up and the threads came back up that it was doing a hcicmd -p -c ‘ pstart” in the background.

We had not made any changes from version 5.8.5 to 6.1.2  in the NetConfig, the xlates, or the tclprocs.

We’ve been told that this has been submitted to R&D, but I’ve not heard anything from them.  Would it be possible to follow up on this.  It’s not happened again, but it’s a time bomb and we don’t know when it will happen again.

[Rob Abbott]

Kevin if you haven’t heard back on your Xtreme case you can ask for an update through Xtreme or give them a call.  We really need to track issues like this through the support system.  Thanks!

Rob Abbott
Cloverleaf Emeritus

[Kevin Scantlan]

Does anyone know of a way to force an abort in a process?  Our issue occurs after a process aborts,  but we’ve not had an abort for over a week (which is good).  It seems like the only way we tell what is doing on, is to “catch it the act”.  I’m not of the belief that it’s not what’s causing the abort that is the issue, but how the engine handles the abort.  We don’t want to turn logging up and just wait for it to happen.  We want to be able to force the issue on the test site that resides on the same server and turn on the logging there.

Thanks.

[Rob Abbott]

If you’re on unix you could kill it with a SIGSEGV signal – e.g.

kill -11

This should cause a panic.  If you use -9 then the process will die “dirty” without a panic.

On Windows you would need to end the process from task manager.

Rob Abbott
Cloverleaf Emeritus

[Kevin Scantlan]

Thanks, Rob!   I used your information and was able to recreate the error!  I did a kill -11 on a process and it PANIC’ed that process.  I brought the process back up and resent a single message into an inbound thread that is in the process.  The message got “stuck” in state 2.  I also got the following in the process log:

[tcl :out :INFO/0:     ccs_ci_4:08/30/2016 09:17:46] 160830 091746.1472566666   ccs_ci_4        CCSED   DFT^P03 20160829110320  2016082911

0320_001                        67-99-39-6()            28368899()                              LEIVA^ELIZABETH^    

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[cmd :cmd :INFO/0:test1_ps04_cmd:08/30/2016 09:17:46] Receiving a command

[cmd :cmd :INFO/0:test1_ps04_cmd:08/30/2016 09:17:46] Command client went away.  Closing connection.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

[pd  :pdtd:ERR /0:     ccs_ci_4:08/30/2016 09:17:46] [0.0.108735679] Delivery to xlate thread test1_ps04_xlate failed. Requeuing msg.

I have been told that there is a “fix” (6.1.2.1).  So once I install that fix, I can recreate the same scenario and see if it is indeed fixed.

Thanks for your help.  It appears we are on our way.

[Bob Schmid]

Might I have a status on this error as we are impacted as well.

running 612 AIX

[Author]

Replies