Alerts: repeat until problem is solved?

[Jeff Shields]

When an alert is triggered, it only alerts once. For example, if a thread goes down we have configured an alert to send an e-mail to support, and also call Tivoli. However, our support guys are concerned that they may miss this single alert.

What they would like is to keep receiving alerts, say every 2 minutes, until the thread is back in an up state. Does anyone have any suggestions on how to achieve this?

Thanks.

Topic

[Tim Hallbauer]

Jeff,

Once an alert condition is met, that particular alert will not fire again.

I believe that the only way to do this is to have multiple alerts set up. An example would be if you have an alert set to trigger if a thread is down for 5, 10, and 15 minutes.

Tim

[garry r fisher]

Hi,

I’m thinking out loud here so please excuse if its a bit rambling.

On your alert create a flag file and run another external job that will monitor the flag every few minutes. If the flag is there send further emails – if its not there then nothing happens.

Create another alert that does the opposite of your original alert i.e. if you alerted on Thread down, then create an alert for thread up. This second alert will check the flag file and delete it and also stop the external monitor process.

Hope this makes sense.

Another option which we use is Nagios (http://www.nagios.ord) that can be setup to respond to your initial email and send continuous reminders until the situation is remedied.

Regards

Garry

[Phil Connolly]

I know this is hopefully late, but forgive me, I just joined.  

Another way to retrigger alerts is to touch the site’s default.alrt file.

[Jeff Shields]

Thanks for the replies.

Yes, I figured in the end that if I change the modification time of the default.alrt file by touching it, it will retrigger the alerts, which is what I need. However, I would like to wait 2 minutes before touching the file, otherwise a flood of alerts would be generated.

I’m just trying to figure out a way to wait 2 minutes without blocking the Monitor Daemon (sleep 120 is no good as we are unable to run hciconnstatus while it’s sleeping).

Jeff

[Phil Connolly]

Jeff,

Experiment with cron.  You may not see what you expect.  For us, even though our default.alrt touch is cron’d to occur once an hour, on the hour, in reality the alert action executes 12 minutes after the hour.  That said, you could still cron these touches for every two minutes (Yeah, kind of bulky, but it may work for you.) for the hours, days, etc. of your choosing.

Cheers!

Phil

[Author]

Replies