Thread Queue Depth

[Randy Colella]

Does anyone know how to extract the queue depth of a thread in a tcl proc, so I can take different actions in the tcl proc based on what the queue depth is.

Topic

[James Cobane]

Randy,

The basic commands to extract the queue depth are:

# This command attaches to the msi (monitor statistics interface)

msiAttach

# This command sets the variable ‘stats’ to the output returned from the msiGetStatSample for the thread specified – ‘mythreadname’

set stats [msiGetStatSample mythreadname]

# This command gets the ob queue depth from the keyed-list in the ‘stats’ variable and puts it into the variable ‘obdataqd’

set obdataqd [keylget stats OBDATAQD]

For more detailed information, look at the msi extensions in the ‘reference guide’.

Hope this helps.

Jim Cobane

Henry Ford Health

[Greg Eriksen]

The only thing I might add, and this is based on suggestions I’ve read in this forum, is that you might want to capture errors, in case you try to attach more than once.

catch {msiAttach}

[Jim Kosloskey]

Randy,

There was a flaw in msi that I don’t think is fixed yet.

Using the thread name in the msiGetStatSample created an error if the thread name began with a numeric. It seems msi decided this must be an index (you can reference the stats either by name or index).

To be safe, I always reference by index and not thread name.

Also be aware I don’t think Quovadx guarantees the msi facility will always behave the same from release to release.

On my 3.2.0 Programmer’s Reference there was a note:

‘These commands and the data they return are primarily for internal use. The interfaces and/or the data may change at any time’.

I do not find such a warning in my 5.2 documentation but I don’t know if that means no problem or if the warning was simply forgotten.

Jim Kosloskey

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Russ Ross]

I’ve mentioned this before but it may be worth mentioning again.

Do not run msiAttach in a site unless

$HciSiteDir/exec/monitorShmemFile

exists; otherwies, you will start to see more and more shared memory issues/conflicts surface over time unecessarily.

Here is an example of how one of our scripts makes that check at the very begining and returns without doing anything if monitorShmemFile does not exist.

Code:

proc oth_check_shmem {} {

   global HciSiteDir
   if { [lsearch -exact [ls $HciSiteDir/exec] monitorShmemFile] == -1 } { return }

   msiAttach
   set msiList [msiTocEntry]

   # the rest of the script has been ommited

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Here is a way to get the que depth in a k-shell script.

Code:

ob_que_depth=`hcimsiutil -dd $thread_name | grep “^OB Data QD”       | awk -F: ‘{print $2}’ | tr -d ‘ ‘`

Which I used in our recycle_thread_alert.ksh script to reduce false alerts as you can see below:

Code:

#!/usr/bin/ksh

# Begin Module Header ==============================================================================
#
#——
# Name:
#——
#
# recycle_thread_alert.ksh
#
#———
# Purpose:
#———
#
# – create a time stamped log entry in an alerts log file called
#   $HCISITEDIR/Alerts/$thread_name.log
# – send e-mail notification to $email_addresses (which could be somebodies pager)
#   with the specified $email_subject
# – recycle the specified $thread_name
#
#——–
# Inputs:
#——–
#
# $1 = email_addresses
# $2 = email_subject
# $3 = thread_name
# $4 = process_name
#
#——-
# Notes:
#——-
#
# Use the alerts configurator to configure alerts to call this script.
#
# Look at /etc/aliases to see all the sendmail email aliases.
#
# Look in directory $HCISITEDIR/Alerts to view the alert messages in the *.log files.
#
# Example of normal usage:
#
#    recycle_thread_alert.ksh                              
#        weekday__p_cbord_adt                              
#        ‘Recycling p_maxsysii because interface is not up’
#        p_cbord_adt                                        
#        adtansils
#
#
# hcimsiutil “Proto Status”:
#
#    0 = thread is dead
#    1 = thread is opening
#    2 = thread is up
#    3 = thread is down
#
#———
# History:
#———
#
# 2000.04.03 Russ Ross
#          – wrote initial version.
#
# 2000.11.24 Russ Ross
#          – modified check to see if the alert is turned off to use the ls command so that
#            symbolic links would casue an alert to be turned off, for example:
#            ln -s /dev/null ib_dms_8053.off
#            this can be used as a visual aid to tell which alerts have been turned off temporarily
#
# 2002.04.02 Russ Ross
#          – modified to start the process and then the thread if the pid file does not exist
#
# 2002.09.12 Russ Ross
#          – modified to use /usr/bin/ksh instead of /bin/ksh
#
# 2003.01.03 Russ Ross
#          – added logic to send out alerts for ib threads
#            * if nothing has been received since recycling 120 seconds ago          
#          – added logic to reduce unecessary alerts by doing the following for non-ib threads
#            * wait 120 seconds after recycling the thread to see if it goes UP
#            * or see if que depth is greater than 200
#          – added logic to explode out the email_addresses and record them in the alert log file
#
# End of Module Header =============================================================================

#———————–
# define input variables
#———————–

email_addresses=$1
email_subject=$2
thread_name=$3
process_name=$4

#————————————–
# define functions local to this script
#————————————–

function recycle_thread {

  if [ ! -f $HCISITEDIR/exec/processes/$process_name/pid ]; then
     hcienginerun -p $process_name
     sleep 5
  fi

  hcicmd -p $process_name -c “$thread_name pstop”
  sleep 5
  hcicmd -p $process_name -c “$thread_name pstart”

}

function log_alert {

     echo “” >>$HCISITEDIR/Alerts/$thread_name.log
     echo ================================================================================ >>$HCISITEDIR/Alerts/$thread_name.log
     echo “” >>$HCISITEDIR/Alerts/$thread_name.log
     date +”%a %b %d %Y %r ($email_subject)” >>$HCISITEDIR/Alerts/$thread_name.log
     echo “” >>$HCISITEDIR/Alerts/$thread_name.log
     echo “Below is a list of who notification of this alert was sent to:” >>$HCISITEDIR/Alerts/$thread_name.log
     echo “If there are no addresses, then only the thread was cycled but no notification was sent!” >>$HCISITEDIR/Alerts/$thread_name.log
     echo “” >>$HCISITEDIR/Alerts/$thread_name.log

}

function send_alert {

     log_alert
     sendmail -bv $email_addresses | awk ‘{print $1}’ >>$HCISITEDIR/Alerts/$thread_name.log
     echo “Subject: $email_subject\n.” | sendmail $email_addresses

}

#———————————————————————
# do not do anything if the alert has been toggled off for this thread
#———————————————————————

# if [ ! -f $HCISITEDIR/Alerts/$thread_name.off ]; then
if [ ! “`ls $HCISITEDIR/Alerts/$thread_name.off 2>/dev/null`” ]; then

  #—————————————–
  # log the fact that an alert got triggered
  #—————————————–
 
  log_alert

  #———————————————–
  # always try to recycle the thread at least once
  #———————————————–

  recycle_thread

  #———————————————————–
  # sleep 120 seconds, then evaluate if need to send out alert
  #———————————————————–

  sleep 120
  proto_status=`hcimsiutil -dd $thread_name | grep “^Proto Status”     | awk -F: ‘{print $2}’ | tr -d ‘ ‘`
  ob_que_depth=`hcimsiutil -dd $thread_name | grep “^OB Data QD”       | awk -F: ‘{print $2}’ | tr -d ‘ ‘`
  ib_last_received=`hcimsiutil -dd $thread_name | grep “^Proto Last Rd” | awk -F: ‘{print $2}’ | tr -d ‘ ‘`

  #————————————————————-
  # send out alert for ib threads
  # if nothing has been received since recycling 120 seconds ago
  #————————————————————-

  if [[ “`echo $thread_name | colrm 3`” = “ib” ]] && [[ “$ib_last_received” = “never” ]]; then
     send_alert
     exit
  fi

  #———————————————————————–
  # send out alert for non-ib threads
  # if thread is still not UP or if outbound que depth is greater than 200
  #———————————————————————–

  if [[ “$proto_status” != “2” ]] || [[ $ob_que_depth > 200 ]]; then
     send_alert
  fi

fi

Russ Ross
RussRoss318@gmail.com

[Author]

Replies