engine hung with no errors or other warnings

Do you have automatic process log cycling turned on?  We had problems with that and backed most of it out.  We are also on CL 5.6 rev2 with AIX 5.3  .  What was the last thing that showed in the process log in question?

Hi Tim,

I have one particular interface with our radiology vendor that has been problematic over the past 3 years.  The problem is that once every 2-3 weeks, the process hangs with no alert and no indication of any problem.  This occurred when we were on version 5.2, and it still occurs now that we are on version 5.6 rev 2.  There has to be something coming across that interface from radiology, but I’ve never been able to figure out the root cause of the process hanging.  So, my solution (work-around) was to put the radiology interfaces in their own site within Cloverleaf.  Within this site there are 2 processes, ARA and monitorARA.  The ARA process contains the actual interface threads, the monitorARA process is used for the purpose of detecting when the ARA process becomes hung.  There are 2 threads within the monitorARA process, a timer thread that creates a simple “hello” message every 30 seconds, and another thread that sends the message to a third thread (fr_monitorARA) via a TCP/IP hop, that is in the actual ARA process.  I have an alert setup to go off when this fr_monitorARA thread has not received a message in more than 45 seconds (since it should be receiving a message every 30 seconds from the monitorARA process).  I then wrote a script that handles all of the shutdown, clean-up, and restart of the ARA process.  This script is set to fire when the alert fires.  This setup has worked well for the past 2 years.  It doesn’t solve the cause of the problem, but it keeps the interfaces going without human intervention.

Tim,

  Did you check your monitord log or as part of your cycling do you do a

  When we first set up our automated monitoring solution we ran into the same problem because the queries to the hcimonitord was making the monitord log file grow too large.  

  Something to look forward to in 5.7, we have automated cycling of log files now.

Hope this helps,

I had the same kind of issue awhile back….Cloverleaf support was pointing me toward a possible problem with a tclproc.  We had two procs that were doing some complex work…..we put the threads into their own process and did alerts etc to handle the hang ups.  We were able to remove those procs after a short period and didn’t have the hang ups any longer.  Just a thought….