Entire site locks up

[Kevin Scantlan]

We tried to bring down a thread and it would not go down. So, we had to bring down kill the process.  As soon as the process was brought back up, messages started flowing into that thread.  In doing research, we discovered that none of the processes in the site were processing any messages during that time.  Then as soon as that one process was brought up, all the rest of the processes started working.  It would seem that the recovery database being locked up by that one process would be the logical candidate for the problem.  However, we don’t see any indications.  The process log does not show any problem.  None of the other process logs (at least the ones I looked at) show any indication either.  

Anyone have any suggestions of where to look?  We run on Cloverleaf 5.6 rev 2 on AIX 5.3  .

Thanks.

Topic

[Tom Rioux]

Kevin,

We had something a little similar happen here to our test server the other day.   I can’t speak to the actual processing of messages since not much testing was going on at the time.  However, trying to bring up or down a thread or process was painstakingly slow.   Nothing in the log files showed any kind of issues.  After talking with our UNIX Administrator, it was learned that some sort of job was running on the DNS server and that is what was causing the issues on our Cloverleaf server.

You may want to talk to your UNIX/Network guys to see if they will own up to doing anything around that time.

Hope this helps…

Tom Rioux

[Robert Gordon]

Had the problem a decade ago, the problems stems from the DNS auto cycle routine, UNIX is more timely than Windows.  We tied the UNIX server to a Windoze server and so the problem began when we cleaned up our host file.  Luckily we could recover quickly since we only uncommented the entries and then rebooted the server.  Also make sure your loopback entry reflects your server ip i.e. you ping 127.0.0.1 you should get your server ip ???.???.???.??? not 127.0.0.1.  for some reason those OS patches sometimes fix or break the problem more and server monitoring software can cause a real headach along with tweaking the kernel parameters and a system reboot.

[Author]

Replies