how to alert when process is hung?

[Gerri Jackson]

hello all!

We have had a couple of instances where a process gets hung, usually in the middle of the night. The process does not panic, so no alert goes out. Since the process is hung up, the other alerts are not triggered, either. Does anyone have a way to monitor and alert when a process is hung, yet the protocol status is UP? Until we upgrade we can not use our Tivoli TMS setup here. Any help would be greatly apprectiated!

Topic

[James Cobane]

Gerri,

What are the symptoms when the process is hung?

[Mark McDaid]

Gerri,

We have a similar problem with one of our processes that gets hung about once every 2 weeks.  The process is called ARA.  My solution was to setup another process called monitorARA.  In this process I have 2 threads, ARAtimer and to_ARAprocess.  The timer thread creates a simple ‘helo’ message every 30 seconds.  This message is routed to the thread to_ARAprocess that sends the message to a new thread that I setup in the ARA process called fr_monitorARA.  The sole purpose of this thread is for receiving and killing the ‘helo’ message.  That way I could setup a last received alert on the fr_monitorARA thread of 45 seconds.  When the ARA process gets hung it stops receiving messages on the fr_monitorARA thread, and 45 seconds later the alert fires, which sends an email, and also runs a script that I wrote to cleanup and restart the hung process.  I’ve been using this setup for about 6 months now and it works great.  Let me know if you need any more details.

[Author]

Replies