busy processes die -possibly linked to automatic log cycling

[Lawrence Williams]

We have several processes that appear to just ‘die’ for some reason.  

It is rather random, but it occurs most frequently with a few particular processes that are some of our busiest processes.

We have automatic log cycling setup for all of our processes, and each time one of these processes dies the last thing in the logfile is a reference to the automatic log cycling – i.e….. “Attempting to cycle output file”

It is also not cycling the log file at the specified size level, we have the size set to 10000 and the logs normally grow much bigger than that before they are cycled.

For instance:  “Automatic cycling of output log files: eoLogFileCycleThreshold:10000 kB logFileSize:40424 kB”

We are running Cloverleaf 5.7 on AIX 5.3

This has been a problem for us for quite some time, and it seems to have started around the same time that we upgraded to 5.7

(we started the automatic log cycling at the same time we upgraded to 5.7, prior to 5.7 we cycled our log files via a cron job).

Topic

[Elijah Park]

Lawrence, we are battling this very same issue. Only we’re running CL 5.6 on AIX 5.3.

After reading the log files and scratching our heads, we decreased LOGCYCLESIZE to 12500 thinking that would work. It didn’t.

Any ideas, Clovertechers?

[Lawrence Williams]

Thank you Daniel for replying (it’s nice to know we’re not the only ones scratching our heads over this issue)……We currently do not have any REV patches installed for 5.7, planning on installing REV2 in the next couple of weeks.  I am hoping there may be something in the patch that will help the situation.

[Richard Hart]

Hi Guys.

I’ve seen this a few times on our test server where were are sending messages as fast as possible. When the log files get to a certain size, the process stops – doesn’t die – just sits there.  Often this is greater that the auto log cycling size and I guess that as we don’t have thread throttling on, the threads working flat tack have a higher priority than the (internal) threads actioning the command queues.

In the most recent case in test I found that a thread stop and start kicked it off again.

I’ve seen this in production where we have Cloverleaf as a web service processing PDF documents and a Bulk Load resulted in huge SMAT files which seemed to confuse the engine and it stopped – again didn’t die! In this case I killed it, manually cycled the SMAT files, restarted and all was OK.

Normally when we run large jobs where we know we’ll get huge SMAT/log files we run the housekeeping tasks (using the Unix ‘at’) more frequently.

[Author]

Replies