Monitoring for Threads down

[john clinkscales]

Has anyone come up with a way to notifiy an operator or page someone if a interface thread is behind or down. We use the depth colors to determine the number of transactions in a queue. However, since we have 55 threads, the operators sometimes miss a thread being down and our physicians are doing EOS (physician order entry) and are waiting on the orders to cross.

Topic

[Steve Carter]

If you’re talking about a single site, it’s fairly simple to do using the msi commands.  First use the msiAttach and then create a thread list by running msTocEntry.  You can then iterate through each thread and obtain stats on each thread by running msiGetStatSample.  This will give you all the information that is available from running hcimsiutil -dd threadName.

You can then page/email based on the criteria that fits your needs.

It gets just a little more complicated if you want to run it across multiple sites, but it is possible.

Let me know if you’d like more detailed information.

Steve

[David Gordon]

Sure, it’s easy.

Get a pager/text-messaging cellphone that is email capable.  Use the Alert configurator to set up your pages, and have it call a script that takes the pager and message as an argument.

For instance, this is our script (KSH script on AIX, proper email address removed obviously):

Code:

#!/bin/ksh
# Replacement for pager-gateway interface for Cloverleaf
# pages.
#
# Convert the first argument to uppercase, and pass the
# second argument to the ‘CurrentMessage’ file, and then
# pass that filename to the mail command as the message.
#
# Cloverleaf messages go to the MTS pager account and are
# also CC’d to the three current Cloverleaf administrators.
#
# Other messages are routed to their respective pagers.
#
# Any other groups are forwarded to myself as an unknown
# group for the purposes of error catching.
#
# Written by David Gordon
# Last revision: March 6th, 2006

pagergroup=$1
typeset -u pagergroup
DATE=`date +%Y%m%d_%H%M`
echo $2 > OutboundMessage

if [ $pagergroup = “CLOVERLEAF” ] ; then
mail -s “Cloverleaf Page” -c “CLGUY1@ADDRESS.COM CLGUY2@ADDRESS.COM CLGUY3@ADDRESS.COM ” PAGER@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "A11" ] ; then mail -s "Cloverleaf Page" PAGER@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "OPERATOR" ] ; then      mail -s "Cloverleaf Automated Email" OPERATOR@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "NOVELL" ] ; then      mail -s "Cloverleaf Page" NOVELL@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "RIVERVIEW" ] ; then      mail -s "Cloverleaf Page" RIVERVIEW@ADDRESS.COM < OutboundMessage elif [ $pagergroup = "RIVERVIEWJAMI" ] ; then      mail -s "Cloverleaf Page" JAMIE@ADDRESS.COM < OutboundMessage else mail -s "Unknown pagergroup!" MY@ADDRESS.COM < OutboundMessage fi echo "Page generated at $DATE" >> /home/pager/pager.log
echo “Page generated for group: $1” >> /home/pager/pager.log
echo “Message: $2″ >> /home/pager/pager.log
echo ” ” >> /home/pager/pager.log

The reason the message is written to a file is that the mail command won’t accept a string as a message input, you need to send it a filename instead.

The advantage of a script like this is that depending on the thread, you can route pages to any one or more pagers that you like.  In this case, we can alert the CL on-call, our ADT admins, operators, Novell admins, or site admins, and it also notifies me if a page comes in for a group that it doesn’t recognize.  It also logs all pages, as well as the time the page was sent.

The script is also nice in that it will accept alerts from multiple sites and/or roots.

So, in the Alert configuration, we have a line like this:

Code:

{VALUE lastr} {SOURCE p_cerner_orders} {MODE actual} {WITH -1} {COMP {> 1800}} {FOR {nmin 5}} {WINDOW {* * 08:00 17:00 * *}} {HOST {}} {ACTION {{exec {/home/pager/pager Cloverleaf “CERNER ORDERS last received a message over 30 minutes ago. THEY may have stalled! (whaprod:pharmnet). [`date`]”}}}}

In this case, it compares the last message read, and if it has not read a message in 30 minutes, it calls the pager script (via the exec command), passes it the ‘Cloverleaf’ argument first, and then the message.

It pages out, and off you go.

[john clinkscales]

Thanks to both of you we will study each solution an try them out.

[Don Gilchrist]

You can also do a e-mail text page directly from alerts.  In the following example e-mail is sent to e-mail address usermail@hosp.org with a CC list in file c:\hci\cphonecc.  %A is the information passed by alert, for example ‘Thread protocol status of CLOVERB_RCV is down’.

We also put a file in a folder c:\hci\Status\intfdown.txt that a second system monitors every 15 minutes to provide a additional level of paging.

We e-mail and text page multiple staff.

blat is a freeware mailer.  All this is on windows.

{VALUE status} {SOURCE {PHSRCV_FWD1 PHSADT_RCV HBPXBCLB_SND PHSRCV_SQL1 IDXRS_SND PHSRCV_FWD3 PHSCHG_RCV HPFRES3_SND HSMRPT_RCV1 HNAM_CHG_FWD Millencg_rcv HNAM_FWD HNAM_CHG HNAMMFN_SND HNAMOR_RCV HNAMAO_SND HNAM_HPF CLOVERB_RCV CBORD_SND HLRMA_SND HELL_SND ROCHE_SND IDXRS_SQL3X quest_res_rcv MILLENQORM_RCV PASSPORT_SND1 HBOCHI_RCV1 HBOCHI_SND1 quest_orm_snd MILLENQORU_SND PASSPORT_RCV11 PASSPORT_RCV111 PHSADTSND_FWD}} {MODE actual} {WITH -2} {COMP {== down}} {FOR once} {WINDOW {* * * * * *}} {HOST {}} {ACTION {{exec {blat – -to usermail@hosp.org -s “%A” -body “%A” -cf c:\hci\cphonecc -q}} {exec {cmd.exe /c copy c:\hci\clpage.txt c:\hci\Status\intfdown.txt}}}}

[Chris Brossette]

I have a script that runs that monitors the information a little differently….it notifies if too many sites are opening and I filter on the count in case you do not care about a certain thread going down at night (regular build-up)…script is below….let me know if you have any questions….

We also monitor if we have not received some activity on an inbound thread….sending system may have a problem but the line may be active.

Let me know if you have any questions….

Chris Brossette

MS Baptist Health Systems

Jackson, MS

601.968.1462

cbrossette@mbhs.org

[Author]

Replies