Threads stuck at initializing

[Brian Sweetland]

I have several QDX 5.3 (rev 5) running on windows 2003 servers.

Every night at midnight, I run a recycle script that saves all the log/smat files to a back drive. Also zero

Topic

[Tom Rioux]

I’m assumming you are using hcimsiutil to zero out the threads.   In my experience, when this is performed, the threads will stay at intializing until messages are sent through.  

Are messages not going through?

Thanks….

Tom Rioux

[James Cobane]

One simple trick to force the status from ‘initializing’ is to simply issue a ‘start’ command on thread.  Even though the thread is already running, the ‘start’ command will force the thread to refresh to it’s normal state.  With that said, normally, the thread will refresh when a  message comes through.

Hope  this helps.

Jim Cobane

Henry Ford Health

[Brian Sweetland]

I don

[Charlie Bursell]

Some of the commands may be trying to execute prior to a previous command completing.  Try a short sleep after some of the commands that take a while or put in statements that make sure they are complete.  Commands like hcienginestop, hcisitectl, hcidbint, etc take a bit to execute

[Rob Abbott]

Brian Sweetland wrote:

hcidbinit -A

Rob Abbott
Cloverleaf Emeritus

[Greg Eriksen]

Brian,

In your list of commands you have “hcisitectl -K” with a capital “K”, which I believe kills all daemons (both the lock manager and monitor daemon).  Then later you have the command “hcisitectl -s m”, which would start the monitor daemon only.  Doesn’t look right to me that you aren’t restarting the lock manager daemon, but then I’m on AIX and maybe that thinking doesn’t apply to Cloverleaf running on Windows 2003?

[Brian Sweetland]

I looked up the lock manager and it appears that’s only for Unix.

I was able to reproduce this in house by running a scheduled task every 10 minutes to recycel the QDX engine. I found 3 times where the threads got stuck at initialization and all 3 times it was a different thread. Here is one sample right before it hung.

[pd  :pdtd:INFO/0:      billing] Throwing startup control switches

[pd  :pdtd:INFO/0:      billing] Throwing OB post-SMS dequeue switch

[icl :icl :DBUG/2:      billing] Sending to thread ‘qdxprocess_xlate’, command ‘pstart billing’

[diag:leak:DBUG/0:      billing] diag imh alloc 0x00A8BC70

[diag:leak:DBUG/0:      billing] diag ptm alloc 0x00A8BC90

[pti :msg :DBUG/1:      billing] Send PTM 0x00A8BC90 from 12 to 1: data 0x00A8BC70

[xlt :thre:DBUG/1:      billing] Notifying processes of new thread

[xlt :thre:DBUG/1:      billing] Open New Thread Event, name: C:gehc-itccgquovadxqdx5.3integrator_brian/witt/NTE

[pd  :thrd:INFO/0:qdxprocess_cmd] Thread reqs auto-started.

[pd  :thrd:INFO/0:qdxprocess_cmd] Starting protocol thread reqs.

[diag:leak:DBUG/0:qdxprocess_cmd] diag psa alloc 0x00A8BCC8

[pti :thre:INFO/1:qdxprocess_cmd] Creating new thread

[pti :thre:DBUG/1:qdxprocess_cmd] Created new pthread, ptid: 0x000000DC

[Scott Folley]

As usual, I am with Charlie on this one.  Most likely the hcidbinit is taking long enough that the rest of the script is executing before it is complete.  If that were to be the case then the default user would still have the database locked when the processes were trying to come up and the process and/or threads would be unable to access the recovery or error database.  At that point they would be locked up tight.  When you run that hcidbinit you should wait for that process to exit before moving to the next step and you should include the -U parameter to specify a different username to use.

Just my .02 but hope that helps…

[Brian Sweetland]

Thanks for the feedback. I

[Scott Folley]

My bad on the -U, that is really for the dbdump…been a while since I have had to do that…getting old and rusty…LOL

One thing I would also ask is why the -R on the hcimsiutil?  Seems to me that all you are really looking for is the -Z to zero out the statistics…shared memory regions and semaphores do not always play as nicely as you would like…using the -Z is still going to cause the threads to go to initializing but you should not have the same issues with removing them and recreating them.

[Brian Sweetland]

I ran these tests over the weekend and still have the same issue. I changed the Sleep to 90 and the hcimsiutil to -Z. Will run more tests and verify.

[Brian Sweetland]

Think I may have found the issue. I can

[David Barr]

I think you need a “catch” around your “exec hostname”.  Exec returns the status of the command not the output.

[Author]

Replies