alert function

This topic has 4 replies, 2 voices, and was last updated 18 years ago by Russ Ross.

Creator

Topic
March 9, 2007 at 12:27 pm #49129
Hongle Sun
Participant
Hello, any one out there can help me out to set alert function up?

Basically we would like to get a email from Cleverleaf interface engine server to alert that all the threads which queued up message over certain amount, says 50… How can i do this, thanks in advance..
Creator

Topic

Viewing 3 reply threads

Author

Replies
- March 9, 2007 at 4:16 pm #60812
  Russ Ross
  Participant
  We are running Cloverleaf 5.2.1P under AIX 5.2, if you are running a different OS like Windows, you likely will use a different sendmail utility for your email/pages.
  
  I would not be surprised if there are already posts on this but it is not much effort for me to copy/paste one of the default.alrt files from one our our sites that shows how we have configured several different alerts, including the one you are interested in.
  
  Code: # Alert config: default.alrt # prologue who: PConnolly date: October 24, 2006 10:39:59 AM CDT type: alrt version: 2.0 end_prologue # {VALUE lastr} {SOURCE jr_global_adt_18310} {MODE actual} {WITH 1} {COMP {>= 900}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__hub_team ‘recycling prod_cbord::jr_global_adt_18310, nothing received for a long time’ jr_global_adt_18310 global_adt}}}} {VALUE lastr} {SOURCE jr_sms_order_18311} {MODE actual} {WITH 1} {COMP {>= 900}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__hub_team ‘recycling prod_cbord::jr_sms_order_18311, nothing received for a long time’ jr_sms_order_18311 sms_order}}}} {VALUE opque} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {> 200}} {FOR {nmin 10}} {WINDOW {* * * * * 6,0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__ob_cbord_adt ‘CBORD Admins – Check ADT interface. More than 200 msgs queued on hub. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE opque} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {> 200}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__ob_cbord_adt ‘CBORD Admins – Check ADT interface. More than 200 msgs queued on hub. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE opque} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {> 200}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__ob_cbord_adt ‘CBORD Admins – Check ADT interface. More than 200 msgs queued on hub. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE opque} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {> 2000}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__hub_team,weekday__ob_cbord_adt ‘CBORD Admins – Check ADT interface. More than 2000 msgs queued on hub. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE opque} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {> 2000}} {FOR {nmin 10}} {WINDOW {* * * * * 6,0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekendy__hub_team,weekend__ob_cbord_adt ‘CBORD Admins – Check ADT interface. More than 2000 msgs queued on hub. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE opque} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {> 2000}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__hub_team,weeknight__ob_cbord_adt ‘CBORD Admins – Check ADT interface. More than 2000 msgs queued on hub. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE opque} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {> 200}} {FOR {nmin 10}} {WINDOW {* * * * * 6,0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__ob_cbord_order ‘CBORD Admins – Check Diet Order interface. More than 200 msgs queued on hub. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE opque} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {> 200}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__ob_cbord_order ‘CBORD Admins – Check Diet Order interface. More than 200 msgs queued on hub. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE opque} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {> 200}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__ob_cbord_order ‘CBORD Admins – Check Diet Order interface. More than 200 msgs queued on hub. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE opque} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {> 2000}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__hub_team,weekday__ob_cbord_order ‘CBORD Admins – Check Diet Order interface. More than 2000 msgs queued on hub. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE opque} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {> 2000}} {FOR {nmin 10}} {WINDOW {* * * * * 6,0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekendy__hub_team,weekend__ob_cbord_order ‘CBORD Admins – Check Diet Order interface. More than 2000 msgs queued on hub. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE opque} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {> 2000}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__hub_team,weeknight__ob_cbord_order ‘CBORD Admins – Check Diet Order interface. More than 2000 msgs queued on hub. prod_cbord:ob_cbord_order’ ob_cbord_orderr sms_order}}}} {VALUE pstat} {SOURCE global_adt} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__hub_team ‘recycling prod_cbord:jr_global_adt_18310 because interface is not up’ jr_global_adt_18310 global_adt}}}} {VALUE pstat} {SOURCE global_adt} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * * * * 6,0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__hub_team ‘recycling prod_cbord:jr_global_adt_18310 because interface is not up’ jr_global_adt_18310 global_adt}}}} {VALUE pstat} {SOURCE global_adt} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__hub_team ‘recycling prod_cbord:jr_global_adt_18310 because interface is not up’ jr_global_adt_18310 global_adt}}}} {VALUE pstat} {SOURCE sms_order} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__hub_team ‘recycling prod_cbord:jr_sms_order_18311 because interface is not up’ jr_sms_order_18311 sms_order}}}} {VALUE pstat} {SOURCE sms_order} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * * * * 6,0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__hub_team ‘recycling prod_cbord:jr_sms_order_18311 because interface is not up’ jr_sms_order_18311 sms_order}}}} {VALUE pstat} {SOURCE sms_order} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__hub_team ‘recycling prod_cbord:jr_sms_order_18311 because interface is not up’ jr_sms_order_18311 sms_order}}}} {VALUE pstat} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__ob_cbord_adt ‘CBORD Admins – Check ADT interface because interface is not up. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE pstat} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 0,6-17 * * 6}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__ob_cbord_adt ‘CBORD Admins – Check ADT interface because interface is not up. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE pstat} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 5-23 * * 0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__ob_cbord_adt ‘CBORD Admins – Check ADT interface because interface is not up. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE pstat} {SOURCE ob_cbord_adt} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__ob_cbord_adt ‘CBORD Admins – Check ADT interface because interface is not up. prod_cbord:ob_cbord_adt’ ob_cbord_adt global_adt}}}} {VALUE pstat} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 7-18 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekday__ob_cbord_order ‘CBORD Admins – Check Diet Order interface because interface is not up. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE pstat} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 0,6-17 * * 6}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__ob_cbord_order ‘CBORD Admins – Check Diet Order interface because interface is not up. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE pstat} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 5-23 * * 0}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weekend__ob_cbord_order ‘CBORD Admins – Check Diet Order interface because interface is not up. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}} {VALUE pstat} {SOURCE ob_cbord_order} {MODE actual} {WITH 1} {COMP {!= up}} {FOR {nmin 10}} {WINDOW {* * 0-6,19-23 * * 1-5}} {HOST {}} {ACTION {{exec {recycle_thread_alert.ksh weeknight__ob_cbord_order ‘CBORD Admins – Check Diet Order interface because interface is not up. prod_cbord:ob_cbord_order’ ob_cbord_order sms_order}}}}
  
  By the way, if you edit the default.alrt file with an editor and comment out entries and then use the GUI the comment lines will get deleted by the GUI .
  
  Here is the recycle_thread_alert.ksh script that gets called to send out email/pages and log the entry to a file. I still need to add a modification to not have simultaneous alerts try to write to the log file at the same time but it will at least give you something tangible to start with.
  
  Code: #!/usr/bin/ksh # Begin Module Header ============================================================================== # #—— # Name: #—— # # recycle_thread_alert.ksh # #——— # Purpose: #——— # # – create a time stamped log entry in an alerts log file called # $HCISITEDIR/Alerts/$thread_name.log # – send e-mail notification to $email_addresses (which could be somebodies pager) # with the specified $email_subject # – recycle the specified $thread_name # #——– # Inputs: #——– # # $1 = email_addresses # $2 = email_subject # $3 = thread_name # $4 = process_name # #——- # Notes: #——- # # Use the alerts configurator to configure alerts to call this script. # # Look at /etc/aliases to see all the sendmail email aliases. # # Look in directory $HCISITEDIR/Alerts to view the alert messages in the *.log files. # # Example of normal usage: # # recycle_thread_alert.ksh # weekday__p_cbord_adt # ‘Recycling p_maxsysii because interface is not up’ # p_cbord_adt # adtansils # # # hcimsiutil “Proto Status”: # # 0 = thread is dead # 1 = thread is opening # 2 = thread is up # 3 = thread is down # #——— # History: #——— # # 2000.04.03 Russ Ross # – wrote initial version. # # 2000.11.24 Russ Ross # – modified check to see if the alert is turned off to use the ls command so that # symbolic links would casue an alert to be turned off, for example: # ln -s /dev/null ib_dms_8053.off # this can be used as a visual aid to tell which alerts have been turned off temporarily # # 2002.04.02 Russ Ross # – modified to start the process and then the thread if the pid file does not exist # # 2002.09.12 Russ Ross # – modified to use /usr/bin/ksh instead of /bin/ksh # # 2003.01.03 Russ Ross # – added logic to send out alerts for ib threads # * if nothing has been received since recycling 120 seconds ago # – added logic to reduce unecessary alerts by doing the following for non-ib threads # * wait 120 seconds after recycling the thread to see if it goes UP # * or see if que depth is greater than 200 # – added logic to explode out the email_addresses and record them in the alert log file # # End of Module Header ============================================================================= #———————– # define input variables #———————– email_addresses=$1 email_subject=$2 thread_name=$3 process_name=$4 #————————————– # define functions local to this script #————————————– function recycle_thread { if [ ! -f $HCISITEDIR/exec/processes/$process_name/pid ]; then hcienginerun -p $process_name sleep 5 fi hcicmd -p $process_name -c “$thread_name pstop” sleep 5 hcicmd -p $process_name -c “$thread_name pstart” } function log_alert { echo “” >>$HCISITEDIR/Alerts/$thread_name.log echo ================================================================================ >>$HCISITEDIR/Alerts/$thread_name.log echo “” >>$HCISITEDIR/Alerts/$thread_name.log date +”%a %b %d %Y %r ($email_subject)” >>$HCISITEDIR/Alerts/$thread_name.log echo “” >>$HCISITEDIR/Alerts/$thread_name.log echo “Below is a list of who notification of this alert was sent to:” >>$HCISITEDIR/Alerts/$thread_name.log echo “If there are no addresses, then only the thread was cycled but no notification was sent!” >>$HCISITEDIR/Alerts/$thread_name.log echo “” >>$HCISITEDIR/Alerts/$thread_name.log } function send_alert { log_alert sendmail -bv $email_addresses | awk ‘{print $1}’ >>$HCISITEDIR/Alerts/$thread_name.log echo “Subject: $email_subject\n.” | sendmail $email_addresses } #——————————————————————— # do not do anything if the alert has been toggled off for this thread #——————————————————————— # if [ ! -f $HCISITEDIR/Alerts/$thread_name.off ]; then if [ ! “`ls $HCISITEDIR/Alerts/$thread_name.off 2>/dev/null`” ]; then #—————————————– # log the fact that an alert got triggered #—————————————– log_alert #———————————————– # always try to recycle the thread at least once #———————————————– recycle_thread #———————————————————– # sleep 120 seconds, then evaluate if need to send out alert #———————————————————– sleep 120 proto_status=`hcimsiutil -dd $thread_name | grep “^Proto Status” | awk -F: ‘{print $2}’ | tr -d ‘ ‘` ob_que_depth=`hcimsiutil -dd $thread_name | grep “^OB Data QD” | awk -F: ‘{print $2}’ | tr -d ‘ ‘` ib_last_received=`hcimsiutil -dd $thread_name | grep “^Proto Last Rd” | awk -F: ‘{print $2}’ | tr -d ‘ ‘` #————————————————————- # send out alert for ib threads # if nothing has been received since recycling 120 seconds ago #————————————————————- if [[ “`echo $thread_name | colrm 3`” = “ib” ]] && [[ “$ib_last_received” = “never” ]]; then send_alert exit fi #———————————————————————– # send out alert for non-ib threads # if thread is still not UP or if outbound que depth is greater than 200 #———————————————————————– if [[ “$proto_status” != “2” ]] || [[ $ob_que_depth > 200 ]]; then send_alert fi fi
  
  Russ Ross
  RussRoss318@gmail.com
- March 9, 2007 at 4:17 pm #60813
  Russ Ross
  Participant
  Here is a visual screen shot of the site for the default.alrt file in my previous post:
  
  Russ Ross
  RussRoss318@gmail.com
- March 9, 2007 at 5:53 pm #60814
  Hongle Sun
  Participant
  Thank you so much. I am new to CLoveleaf.
  
  My question next would be, inorder to use the alert function, what should i dont or turn on the Alert , another quesiton is i saw the recycle_thread_alert.ksh, where should i put this script in order to call?
  
  Thanks
- March 9, 2007 at 7:25 pm #60815
  Russ Ross
  Participant
  The default.alrt file is loaded when you start the monitor daemon, which has to be running for you to look at interfaces in the netmonitor GUI.
  
  You can put recycle_thread_alerts.ksh in a directory of your choosing so long as it is in the path for hci.
  
  In my case I have a site just for global objects called mda_global and in that directory I created a scripts directory as follows
  
  /quovadx/qdx5.2/integrator/mda_global/scripts
  
  then I modified /quovadx/qdx5.2/integrator/bin/hcisetenv so my hci path includes the mda_global scripts directory as follows:
  
  /quovadx/qdx5.2/integrator/prod_cbord/bin
  
  /quovadx/qdx5.2/integrator/prod_cbord/scripts
  
  /quovadx/qdx5.2/integrator/bin
  
  /quovadx/qdx5.2/integrator/mda_global/scripts
  
  /quovadx/qdx5.2/integrator/contrib
  
  /quovadx/qdx5.2/integrator/sbin
  
  /quovadx/qdx5.2/integrator/dbms/bin
  
  /quovadx/qdx5.2/integrator/tcl/bin
  
  /quovadx/qdx5.2/integrator/clgui/bin
  
  /quovadx/qdx5.2/integrator/clgui/java/bin
  
  /quovadx/qdx5.2/integrator/usercmds
  
  /usr/bin
  
  /etc
  
  /usr/sbin
  
  /usr/ucb
  
  /home/hci/bin
  
  /usr/bin/X11
  
  /sbin
  
  .
  
  /usr/local/bin
  
  /usr/local/scripts
  
  /quovadx/qdx5.2/integrator/kshlib
  
  The reason I have a site for all global objects is so that cloverleaf upgrades will not blow them away and they will go through any upgrade conversion process.
  
  I sympathize you are new and may have more questions than answers but I don’t really want to go down the rabbit hole too far but I wish you success with your learning process.
  
  I would even offer to talk to you offline but I will not be available from 3/10 – 3/18 and I suspect you will be well on your way by then.
  
  Russ Ross
  RussRoss318@gmail.com
Author

Replies

Viewing 3 reply threads

The forum ‘Cloverleaf’ is closed to new topics and replies.