Automatic log rotation

[Vaughn Skinner]

I recently turned on log rotation at 200M.  Sometimes when the logs are rotated, the process crashes.  It appears to work sometimes.  Has anyone seen this before?

From the process exit_log:

Abnormal exit – Cloverleaf software panic

From the process log:

[prod:prod:DBUG/1:ld_results_supergroup_in:12/17/2009 17:04:00] Automatic cycling of output log files: eoLogFileCycleThreshold:200000 kB logFileSize:200005 kB errFileSize:0 kB

[prod:prod:INFO/0:ld_results_supergroup_in:12/17/2009 17:04:00] HCI_EOCYCLE_TRIES = 200.

[prod:prod:WARN/0:ld_results_supergroup_in:12/17/2009 17:04:00] Attempting to cycle output file.

Topic

[Russ Ross]

One of our prodcution standards for about the last 6 years requires all cloverleaf processes have automatic log cycling turned on.

We typically use 50,000 (about 50M) limits and have yet to witness the problem you described over the last 6 years I would estimate.

Our automatic 50M limit rarely gets triggered becuase we have a cron entry proactively recycle all log files daily near midnight.

I have a couple of thoughts or more like questions.

Is there enough free space in your filesystem to cycle the log file(s)?

Do you have any proactive daily log cycling scheduled and does that ever have a similar problem?

We only use EO configy for short periods for troubleshooting but I’m curious if EO config all is being used on any of the processes that are crashing?

I ask this becuase that would generate excessive I/O to the log file that might burden the automatic log cycling.

Russ Ross
RussRoss318@gmail.com

[Vaughn Skinner]

We do not do a nightly log rotation.

We have adequate space.

We have EO Config setup with engines at Info level 0 and msg logging at info+debug and all levels.

Are you thinking that it might crash if log rotating and a new log messages need to be written?

[Shibu Mathew]

Hi Russ,

Do I set the 50 MB limit to “Automatic Log Cycling” under Process configuration (process->configure), OR,  to Automatic Log Cycling Threshold in “Message Archiving” (Options-> Site Preferences).

I also see “Max Logs Total Size” in Log History under options->Site Options->Site Preferences-> Log History.

Thank you.

[Mark Thompson]

Vaughn,

Do you have LogHistory enabled on your sites?

We cycle log files for 60 processes every 4 hours.  Using LogHistory to compress our log files resulted in occasional (1 every few days) process crashes during log file cycling.

We disabled LogHistory and implemented a cron job to sweep all the process directories every few minutes for log.old and err.old files, moving and compressing them.  It is possible for this method to miss log files if a process is restarted repeatedly before the .old files are archived.

- Mark Thompson
HealthPartners

[Vaughn Skinner]

Mark,

I do.  I will try disabling that and see if the crashes stop.

Thank you

[Russ Ross]

Mark, thanks for sharing that concern about LogHistory becuase I’ve been considering using it instead of my daily cron cycling.

Shibu Mathew wrote

Code:

Hi Russ,
Do I set the 50 MB limit to “Automatic Log Cycling” under Process configuration (process->configure),
OR,  to Automatic Log Cycling Threshold in “Message Archiving” (Options-> Site Preferences).
I also see “Max Logs Total Size” in Log History under options->Site Options->Site Preferences-> Log History.

NetConfig -> Process in menu choices at top-> check box Automatic Log Cycling -> Threshold in Kilobytes = 50000

Russ Ross
RussRoss318@gmail.com

[Shibu Mathew]

Russ,

Thanks.

[Vaughn Skinner]

I have turned LogHistory back on and disabled the compression.  I haven’t had any more trouble this week.  I can enable the compression as a cron job in the middle of the night.

[Kevin Scantlan]

The cycling works for every process with the exception of one.  It’s a very busy one…. and it constantly busy.  We receive over 17,000 messages every hour, with about 2,900 messages every 10 minutes…. 7 x 24.  We feel that when the log size goes over the threshold, a trigger is set to cycle the logs, but does not have high enough priority to interrupt the process to cycle.  Instead it waits for a pause in the action, which it never gets.  However, when we execute a hcicmd either manually or from a script, it cycles the logs twice:  once is an automated cycle followed immediatly with the hcicmd cycle.  We feel that our process aborted because the log file had grown to over 300M and had to do a compression, and then was followed by an hcicmd and tried to cycle again, but was not done with the compression.  

I e-mailed support, got a ticket number, gave them more information, but have not heard back in since the end of last week.  So, we put in a script that checks that process every 15 minutes to see if the log file size is over 20M and if it is, we cycle the logs.  We only do this with the process that’s having the problem and this seems to have circumvented the problem.

[Vaughn Skinner]

My problems went away as soon as I disabled compression.  Now I use a cron job in the middle of the night to compress any .log files.  This lets me use the regular log rotation settings within cloverleaf.

[Baron Matthews]

On that same subject, I have seen panic’s crop up when log files get quite large and then you try to do some type of operation with them.  Looks like the compression operation was your culprit (as you found).  I had an environment that I forgot to enable any of the auto log type features, then, when I set them up after not having them on for quite some time, I got a very similar error.  Just keep those sizes small!

[Author]

Replies