2 Gig log file then process crashes

Homepage Clovertech Forums Read Only Archives Cloverleaf Cloverleaf 2 Gig log file then process crashes

  • Creator
    Topic
  • #50068
    Andrew Deters
    Participant

    We receive these errors from one of our feeds.  Typically this message is repeated several thousand times per minute.  Eventually (roughly 2 hours), the log file and error file grow to 2Gig in size each and the process crashes.  That leads to the logical volume running out of space when those crash files are copied to the errors directory.

    [prod:prod:WARN/0:   engine_cmd:05/21/2008 23:45:17] Output log file cycled

    [pdl :PDL :ERR /0:          FEED1:05/22/2008 01:46:38] read failed: Connection timed out

    [pdl :PDL :ERR /0:          FEED1:05/22/2008 01:46:38] read returned error 110 (Connection timed out)

    Followed by this message a little later (probably when the connection is restored).

    [pdl :PDL :ERR /0:         FEED1:05/22/2008 19:28:37] read returned error 0 (Success)

    Whatever is happening at this point it is telling me the error message is 0 “Success”.  

    These feeds are connected via a vpn to a vendor.  The EOConfig for these is disable_all.  I know it is a protocol error but am at a loss in how to troubleshoot the best way to fix the problem.  Has anyone every seen this type of behavior?  It does not happen at the same time.

Viewing 4 reply threads
  • Author
    Replies
    • #64754
      James Cobane
      Participant

      Andrew,

      Why don’t you just turn-on automatic Log cycling on the process?  Set the size (in KB) to what seems reasonable, and you should be good to go.

      Jim Cobane

      Henry Ford Health

    • #64755
      Kevin Kinnell
      Participant

      I’d definitely do as Jim suggested, but I’d look pretty closely at that thread

      setup.  We had a similar situation with fast growing error logs.  In that case,

      there were multiple problems, but what allowed us to debug it was to

      increase the timeout for ACKs, which kept things sane enough that we could

      gather enough information to blame the other side of the connection.  😉

      What happens when you “stutter” the thread (bring it up and almost

      immediately stop it again)?

    • #64756
      Andrew Deters
      Participant

      I’ll set up the automatic log cycling.  They currently cycle once per day.  I tried to stop and start the feed repeatedly and do not recieve the results described above.  The feeds act as expected.  

      When i’m able to bring down the VPN tunnel, I’ll attempt stopping and starting the tunnel and see if that doesn’t produce the above results.  

      Thanks

    • #64757
      Paul Johnston
      Participant

      Andrew,

      We have run into the exact same issue a while back . It crashed our system a couple of times.

      We developed a UNIX script that examines the size of the Error file . When is gets to a certain size we bound the process.

      I like Jim’s solution also .

    • #64758
      Russ Ross
      Participant

      It is not surprising that it crashes when the log file size reaches 2 GB’s.

      Typically that is the maximum file size defined in the /etc/security/limits file.

      Here at MD Anderson Cancer Center it is a mandatory in-house standard for all production processes to have automatic log cycling implemented.

      Even if the interface does not have any problems all it takes is to turn on EO config output and forget to turn it off, and our log file will grow to 2 GB long before your daily cycling of our log files get invoked via cron.

      Russ Ross
      RussRoss318@gmail.com

Viewing 4 reply threads
  • The forum ‘Cloverleaf’ is closed to new topics and replies.

Forum Statistics

Registered Users
5,042
Forums
28
Topics
9,200
Replies
34,021
Topic Tags
267