All threads in a process in "initializing" mode

[Sam Craig]

Running Cloverleaf 5.5 Rev1 on 5.3 AIX.

Recently, I have a process in out test region that seems to get “hung” in initializing mode during the nightly log archiving activity that puts all the days logs into the savefiles directory.  Using the IMArchive5.sh & IMArchive5.tcl to do the archiving.

I have been using this code for many years and have never seen these results.

All the threads have this output when I check the status.

Connection : midasHPF_ob

Reported at: Fri Jul 23 10:25:59 EDT 2010

Status   : initializing   Started: Wed Dec 31 19:00:00 EST 1969

Msgs in  : 0              Last Rd: Never

Bytes in : 0

Msgs out : 0              Last Wt: Never

Bytes out: 0

Proto Err: Never

Error Msg:

Weirdest thing is that it says all threads were started Wed Dec 31 19:00:00 EST 1969.

Trust me, that is incorrect. 😯

I have checked the IMArchive5.log, process logs, and error logs.

Can’t find anything that says anything went wrong during archive process.

Anybod else seen this issue or any suggestions on where to look to find log with info about this event.

Good news is that all I have to do to fix is stop and restart process.

Thx,

Sam

Topic

[Michael Hertel]

This is common until the next activity happens. (usually the next message through will take care of it)

If this is production and you know messages are flowing, then I’d be concerned. Most likely, your NetMonitor is lying to you and needs to be refreshed (ctrl-r)

Additionally, you don’t have to stop/start. You can make “something” happen in one thread in the process. My favorite is to click ‘release’ on an outbound thread. Then all the thread’s statuses in that process wake up.

[Tom Rioux]

I see this all the time in my test environment.   You don’t even have to bounce the process.   The status will change for the threads in the process once a message comes through.    

We are on 5.6 and not really sure what is causing it.

Tom

[Rob Abbott]

Status on a running thread will change to “initializing” if statistics are reset using “hcimsiutil -Z”.  

It will stay in this status until an event happens on that thread or the process that owns the thread.  

If you have an automated job that is zeroing statistics, this is probably the cause.

Rob Abbott
Cloverleaf Emeritus

[Russ Ross]

I too came to realize that cycling the thread stats via hcimsiutil -Z will leave the interface in an initializing state until a message comes across.

Of course I did not enjoy learning this as prodcution on-call alerts kept going off in the midle of the night shortly after the nightly zero stats script ran and by the time you looked at it everything seemed okay other than you kept loosing sleep.

Now what I do to prevent this anymore is to cycle the stats nightly followed by cycle the log files right after that.

Cycling the log file like any other thread action seems to put the thread back into the desired UP state.

Russ Ross
RussRoss318@gmail.com

[Sam Craig]

Thanks for all the info….. 😀

Just seems weird that I have not seen this happen to a process since I started working with Cloverleaf about 6 years ago.

I have other processes in our test region that don’t have any activity overnight, after the automated job runs, and they all go back to either opening or up state.

Thanks,

Sam

[Author]

Replies