Reply To: Monitoring for Threads down

[David Gordon]

Sure, it’s easy.

Get a pager/text-messaging cellphone that is email capable.  Use the Alert configurator to set up your pages, and have it call a script that takes the pager and message as an argument.

For instance, this is our script (KSH script on AIX, proper email address removed obviously):

Code:

#!/bin/ksh
# Replacement for pager-gateway interface for Cloverleaf
# pages.
#
# Convert the first argument to uppercase, and pass the
# second argument to the ‘CurrentMessage’ file, and then
# pass that filename to the mail command as the message.
#
# Cloverleaf messages go to the MTS pager account and are
# also CC’d to the three current Cloverleaf administrators.
#
# Other messages are routed to their respective pagers.
#
# Any other groups are forwarded to myself as an unknown
# group for the purposes of error catching.
#
# Written by David Gordon
# Last revision: March 6th, 2006

pagergroup=$1
typeset -u pagergroup
DATE=`date +%Y%m%d_%H%M`
echo $2 > OutboundMessage

if [ $pagergroup = “CLOVERLEAF” ] ; then
mail -s “Cloverleaf Page” -c “CLGUY1@ADDRESS.COM CLGUY2@ADDRESS.COM CLGUY3@ADDRESS.COM ” PAGER@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "A11" ] ; then mail -s "Cloverleaf Page" PAGER@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "OPERATOR" ] ; then      mail -s "Cloverleaf Automated Email" OPERATOR@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "NOVELL" ] ; then      mail -s "Cloverleaf Page" NOVELL@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "RIVERVIEW" ] ; then      mail -s "Cloverleaf Page" RIVERVIEW@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "RIVERVIEWJAMI" ] ; then      mail -s "Cloverleaf Page" JAMIE@ADDRESS.COM < OutboundMessage else mail -s "Unknown pagergroup!" MY@ADDRESS.COM < OutboundMessage fi echo "Page generated at $DATE" >> /home/pager/pager.log
echo “Page generated for group: $1” >> /home/pager/pager.log
echo “Message: $2″ >> /home/pager/pager.log
echo ” ” >> /home/pager/pager.log

The reason the message is written to a file is that the mail command won’t accept a string as a message input, you need to send it a filename instead.

The advantage of a script like this is that depending on the thread, you can route pages to any one or more pagers that you like.  In this case, we can alert the CL on-call, our ADT admins, operators, Novell admins, or site admins, and it also notifies me if a page comes in for a group that it doesn’t recognize.  It also logs all pages, as well as the time the page was sent.

The script is also nice in that it will accept alerts from multiple sites and/or roots.

So, in the Alert configuration, we have a line like this:

Code:

{VALUE lastr} {SOURCE p_cerner_orders} {MODE actual} {WITH -1} {COMP {> 1800}} {FOR {nmin 5}} {WINDOW {* * 08:00 17:00 * *}} {HOST {}} {ACTION {{exec {/home/pager/pager Cloverleaf “CERNER ORDERS last received a message over 30 minutes ago. THEY may have stalled! (whaprod:pharmnet). [`date`]”}}}}

In this case, it compares the last message read, and if it has not read a message in 30 minutes, it calls the pager script (via the exec command), passes it the ‘Cloverleaf’ argument first, and then the message.

It pages out, and off you go.