- This topic has 11 replies, 6 voices, and was last updated 14 years, 4 months ago by
.
-
Topic
-
I recently turned on log rotation at 200M. Sometimes when the logs are rotated, the process crashes. It appears to work sometimes. Has anyone seen this before? From the process exit_log:
Abnormal exit – Cloverleaf software panic
From the process log:
[prod:prod:DBUG/1:ld_results_supergroup_in:12/17/2009 17:04:00] Automatic cycling of output log files: eoLogFileCycleThreshold:200000 kB logFileSize:200005 kB errFileSize:0 kB
[prod:prod:INFO/0:ld_results_supergroup_in:12/17/2009 17:04:00] HCI_EOCYCLE_TRIES = 200.
[prod:prod:WARN/0:ld_results_supergroup_in:12/17/2009 17:04:00] Attempting to cycle output file.
-
Replies
-
-
One of our prodcution standards for about the last 6 years requires all cloverleaf processes have automatic log cycling turned on.
We typically use 50,000 (about 50M) limits and have yet to witness the problem you described over the last 6 years I would estimate.
Our automatic 50M limit rarely gets triggered becuase we have a cron entry proactively recycle all log files daily near midnight.
I have a couple of thoughts or more like questions.
Is there enough free space in your filesystem to cycle the log file(s)?
Do you have any proactive daily log cycling scheduled and does that ever have a similar problem?
We only use EO configy for short periods for troubleshooting but I’m curious if EO config all is being used on any of the processes that are crashing?
I ask this becuase that would generate excessive I/O to the log file that might burden the automatic log cycling.
Russ Ross
RussRoss318@gmail.com -
-
Hi Russ,
Do I set the 50 MB limit to “Automatic Log Cycling” under Process configuration (process->configure), OR, to Automatic Log Cycling Threshold in “Message Archiving” (Options-> Site Preferences).
I also see “Max Logs Total Size” in Log History under options->Site Options->Site Preferences-> Log History.
Thank you.
-
Vaughn,
Do you have LogHistory enabled on your sites?
We cycle log files for 60 processes every 4 hours. Using LogHistory to compress our log files resulted in occasional (1 every few days) process crashes during log file cycling.
We disabled LogHistory and implemented a cron job to sweep all the process directories every few minutes for log.old and err.old files, moving and compressing them. It is possible for this method to miss log files if a process is restarted repeatedly before the .old files are archived.
- Mark Thompson
HealthPartners -
-
Mark, thanks for sharing that concern about LogHistory becuase I’ve been considering using it instead of my daily cron cycling.
Shibu Mathew wrote
Code:Hi Russ,
Do I set the 50 MB limit to “Automatic Log Cycling” under Process configuration (process->configure),
OR, to Automatic Log Cycling Threshold in “Message Archiving” (Options-> Site Preferences).
I also see “Max Logs Total Size” in Log History under options->Site Options->Site Preferences-> Log History.NetConfig -> Process in menu choices at top-> check box Automatic Log Cycling -> Threshold in Kilobytes = 50000
Russ Ross
RussRoss318@gmail.com -
-
-
The cycling works for every process with the exception of one. It’s a very busy one…. and it constantly busy. We receive over 17,000 messages every hour, with about 2,900 messages every 10 minutes…. 7 x 24. We feel that when the log size goes over the threshold, a trigger is set to cycle the logs, but does not have high enough priority to interrupt the process to cycle. Instead it waits for a pause in the action, which it never gets. However, when we execute a hcicmd either manually or from a script, it cycles the logs twice: once is an automated cycle followed immediatly with the hcicmd cycle. We feel that our process aborted because the log file had grown to over 300M and had to do a compression, and then was followed by an hcicmd and tried to cycle again, but was not done with the compression.
I e-mailed support, got a ticket number, gave them more information, but have not heard back in since the end of last week. So, we put in a script that checks that process every 15 minutes to see if the log file size is over 20M and if it is, we cycle the logs. We only do this with the process that’s having the problem and this seems to have circumvented the problem.
-
-
On that same subject, I have seen panic’s crop up when log files get quite large and then you try to do some type of operation with them. Looks like the compression operation was your culprit (as you found). I had an environment that I forgot to enable any of the auto log type features, then, when I set them up after not having them on for quite some time, I got a very similar error. Just keep those sizes small!
-
- The forum ‘Cloverleaf’ is closed to new topics and replies.
