Alerts: repeat until problem is solved?

Clovertech Forums Read Only Archives Cloverleaf Cloverleaf Alerts: repeat until problem is solved?

  • Creator
    Topic
  • #47687
    Jeff Shields
    Participant

      When an alert is triggered, it only alerts once. For example, if a thread goes down we have configured an alert to send an e-mail to support, and also call Tivoli. However, our support guys are concerned that they may miss this single alert.

      What they would like is to keep receiving alerts, say every 2 minutes, until the thread is back in an up state. Does anyone have any suggestions on how to achieve this?

      Thanks.

    Viewing 4 reply threads
    • Author
      Replies
      • #56450
        Tim Hallbauer
        Participant

          Jeff,

          Once an alert condition is met, that particular alert will not fire again.

          I believe that the only way to do this is to have multiple alerts set up. An example would be if you have an alert set to trigger if a thread is down for 5, 10, and 15 minutes.

          Tim

        • #56451
          garry r fisher
          Participant

            Hi,

            I’m thinking out loud here so please excuse if its a bit rambling.

            On your alert create a flag file and run another external job that will monitor the flag every few minutes. If the flag is there send further emails – if its not there then nothing happens.

            Create another alert that does the opposite of your original alert i.e. if you alerted on Thread down, then create an alert for thread up. This second alert will check the flag file and delete it and also stop the external monitor process.

            Hope this makes sense.

            Another option which we use is Nagios (http://www.nagios.ord) that can be setup to respond to your initial email and send continuous reminders until the situation is remedied.

            Regards

            Garry

          • #56452
            Phil Connolly
            Participant

              I know this is hopefully late, but forgive me, I just joined.  

              Another way to retrigger alerts is to touch the site’s default.alrt file.

            • #56453
              Jeff Shields
              Participant

                Thanks for the replies.

                Yes, I figured in the end that if I change the modification time of the default.alrt file by touching it, it will retrigger the alerts, which is what I need. However, I would like to wait 2 minutes before touching the file, otherwise a flood of alerts would be generated.

                I’m just trying to figure out a way to wait 2 minutes without blocking the Monitor Daemon (sleep 120 is no good as we are unable to run hciconnstatus while it’s sleeping).

                Jeff

              • #56454
                Phil Connolly
                Participant

                  Jeff,

                  Experiment with cron.  You may not see what you expect.  For us, even though our default.alrt touch is cron’d to occur once an hour, on the hour, in reality the alert action executes 12 minutes after the hour.  That said, you could still cron these touches for every two minutes (Yeah, kind of bulky, but it may work for you.) for the hours, days, etc. of your choosing.

                  Cheers!

                  Phil

              Viewing 4 reply threads
              • The forum ‘Cloverleaf’ is closed to new topics and replies.