Connection bounce is resetting alerts

[John Zalesak]

I am new to alerts but there are some setup already in our system.  The employees who set them up are no longer here so I am on my own.

Here is what I want to do.

I want to have an alert on last received.  I want to bounce the thread if it did not get something in the last 30 minutes.  If it did not get something in 95 minutes, I want an email.

The idea is bounce a couple of times, if still no messages, alert someone.

My problem is that I think the short time alert bounce is resetting the longer time alert begin time and it never goes off.

For example: here is what I set up in a test site:

First Bounce Alert

Type: last receive

Source: Inbound_side

Source Count: any

Comparing: >= 60

Duration: once

Second Email Alert

Type: last receive

Source: Inbound_side

Source Count: any

Comparing: >= 120

Duration: once

When I run it,  the 60 second alert continually goes off but the 120 second alert never does.

Any ideas on how to bring my idea into reality??

Thanks in advance for your responses.

Topic

[Robert Kersemakers]

Hi John,

It looks that way, yes. The first alert always fires, so maybe the second one doesn’t because the ‘trigger’ for this thread is reset.

Have you tried putting the second alert before (ie above) the first alert in the Alert Configuration file? It should cause the second alert to fire (if >120) and if not, the first alert should still fire (if >60).

Just a wild guess here…

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[John Zalesak]

Robert

Thanks for your idea.  I will give it a try and let you know.

[John Zalesak]

Robert,

I tried your idea first.

Alerts in this order

NUMBER 1

Type: last receive

Source: Inbound_side

Source Count: any

Comparing: >=75

Duration: Once

NUMBER 2

Type: last receive

Source: Inbound_side

Source Count: any

Comparing: >=60

Duration: Once

-> All I ever got was Number 2 (60 secs)

Then I tried

NUMBER 1

Type: last receive

Source: Inbound_side

Source Count: any

Comparing: >=1

Duration: nsec 75

NUMBER 2

Type: last receive

Source: Inbound_side

Source Count: any

Comparing: >=1

Duration: nsec 60

The only one I got to fire was NUMBER 2 (61 sec)

I am really grabbing a straws here.

Can anyone point me in the correct direction??

Thanks!

[John Zalesak]

Here is a little more info.  I checked my hcimonitord file.  Looks OK to me.  The longer alert (Alert #12) never fires.

Any comments or assistance is greatly appreciated.

[aler:aler:INFO/0:  hcimonitord:11/13/2008 11:15:02] New alert #12:

{VALUE lastr} {SOURCE {Inbound_side }} {MODE actual} {WITH -2} {COMP {>= 1}} {FOR {nsec 75}} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – %A. Connection bounced.”}}}}

[aler:aler:INFO/0:  hcimonitord:11/13/2008 11:15:02] can’t read “HCISITEDIR”: no such variable

[aler:aler:INFO/0:  hcimonitord:11/13/2008 11:15:02] New alert #13:

{VALUE lastr} {SOURCE {Inbound_side }} {MODE actual} {WITH -2} {COMP {>= 1}} {FOR {nsec 60}} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – %A. Connection bounced.”}}}}

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:16:07] Alert #13 triggered.

alert: {VALUE lastr} {SOURCE {Inbound_side }} {MODE actual} {WITH -2} {COMP {>= 1}} {FOR {nsec 60}} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – %A. Connection bounced.”}}}}

action: IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – Thread last inbound message received time of Inbound_side has been more than or equal to 1for 60 seconds. Connection bounced.” &

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:16:07] Completed Cascade Actions

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:17:22] Alert #13 triggered.

alert: {VALUE lastr} {SOURCE {Inbound_side }} {MODE actual} {WITH -2} {COMP {>= 1}} {FOR {nsec 60}} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – %A. Connection bounced.”}}}}

action: IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – Thread last inbound message received time of Inbound_side has been more than or equal to 1for 60 seconds. Connection bounced.” &

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:17:22] Completed Cascade Actions

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:18:37] Alert #13 triggered.

alert: {VALUE lastr} {SOURCE {Inbound_side }} {MODE actual} {WITH -2} {COMP {>= 1}} {FOR {nsec 60}} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – %A. Connection bounced.”}}}}

action: IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – Thread last inbound message received time of Inbound_side has been more than or equal to 1for 60 seconds. Connection bounced.” &

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:18:37] Completed Cascade Actions

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:19:53] Alert #13 triggered.

alert: {VALUE lastr} {SOURCE {Inbound_side }} {MODE actual} {WITH -2} {COMP {>= 1}} {FOR {nsec 60}} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – %A. Connection bounced.”}}}}

action: IMalerts_jtz.sh sitejtz tst_pro Inbound_side IM_Alert_Last_Recd “$HCISITEDIR – Thread last inbound message received time of Inbound_side has been more than or equal to 1for 60 seconds. Connection bounced.” &

[aler:aler:WARN/0:  hcimonitord:11/13/2008 11:19:53] Completed Cascade Actions

[Michael Hertel]

Yes you are grabbing.

By bouncing you are resetting the stats on the thread so

your email will never fire.

[John Zalesak]

So what is the trick.

I would like to have it try and fix it self a few times before I get woken up at 3 am.

Is there another way to come at this problem???

[Michael Hertel]

You could write an alert script that reads/writes to a file

which you could extract previous stats.

If it were me, I’d email with the first alert and when I see multiple back to back emails I would know there is an issue.

You might even be able to have the alert script fire off and look again in x amount of time to see if it fixed itself.

I’ll post one of our scripts in a minute.

[Michael Hertel]

Here is part of our script:

Look at the outbound qdepth portion to see the sleep statement.



#!/hci/root/bin/hcitcl
set ts [clock format [clock seconds]]
[code]

#!/hci/root/bin/hcitcl
set ts [clock format [clock seconds]]

[John Zalesak]

Michael,

Thanks for the script.  I go to Tcl class next week so maybe it will make a little more sense when I get back…. If I pay attention!

Currently, we do get the email on the first alert, but after being woken up at 3am for a couple of days only to say, its the first one go back to bed.  You starting thinking about doing something different.

Our 1st idea (simple) is to

Set alert to bounce if none received in 45 minutes

Set alert to send e-mail to blackberry (and get me out of bed) if the tread is opening for 30 minutes.

Hopefully the bounce fixes the problem.  If after the bounce, a connection can not be made in 30 minutes, its time to get woken up.

Our 2nd idea (more complex) is to

Set alert to bounce in none received in say 15 minutes

The script that bounces should write a time stamp to a log file.

The script will also read the log file, do an analysis – say maybe 4 bounces in a row-, and if need be -> send an email to my blackberry to wake me up.

Any comments / suggestions would be greatly appreciated.

[Michael Hertel]

Good luck with the class  😀

[John Zalesak]

thanks

[James Cobane]

John,

You could utilize the counter functions provided by Cloverleaf (i.e. CtrNextValue, CtrResetValue ) within a tcl script to keep a counter of how many times the first alert fires.  You could run this script on your 30-minute Alert; if the counter value is >= your desired number (i.e. 3), then you could reset the counter and trigger the e-mail within the script via an ‘exec’ command.

Also, I don’t believe the order of the alerts has any effect; it’s all based on the defined conditions.  Order would likely only play a role if two alerts had the same condition defined.

Hope this helps.

Jim Cobane

Henry Ford Health

[John Zalesak]

James – Thanks for you comments

I agree, in my testing, the order of the alerts has no effect.

Thanks for the counter functions in Cloverleaf.  I had no idea they were there.  Where do I find out about them???  Are there counters that will tell us no messages over the last 3 bounces (Alerts) ???

We thought of a similar idea with our own counter stored in a file.  Our problem was how do you differentiate between 3 bounces in the last 90 minutes versus 3 bounce in the last week.  

Thanks again.

[James Cobane]

John,

The counter functions are documented in the ‘Reference Manual’ under Tcl extensions, Counter Commands.  The counter commands allow you to create your own counter files for whatever use you desire (i.e. creating/maintaining sequence numbers, etc).  To determine if the count is for the last 90 minutes vs. the past week, you’ll probably have to store the date/time info off into a file to use for comparison.

Jim Cobane

Henry Ford Health

P.S.  Have fun in your tcl class!

[Author]

Replies