Reset and Collect Statistics into a DB – TCL – hcimsiutil

[Erik Mueller]

Hi,

I’m wondering if anybody has written something similar to this already that is willing to share it with me.

I’m wanting to collect the statistics for the threads on a regular basis and then reset them e.g. once a week, I collect the stats and then reset them.

Has anybody written a TCL that will extract the statistics for a thread and store them in a SQLite DB (or a text file)?

I’m not interested in all the statistics, just the Msgs In, Msgs Out, Xlated and Failed.

Anything out there would be appreciated,

Thanks,

Erik

Topic

[Jim Kosloskey]

I have an old proc that dumps the entire msi stats to a delimited file whiich could then be imported to any tool allowing importation of a delimited file (DB included).

Email me and I will send it – you can do with it what you want.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Elisha Gould]

I’d actually be interested in this as well. I was just thinking about this the other day to help with statistics when users ask about volumes throughout the day, and also to help detect potential issues on an interface.

[Rob Lindsey]

I have something that writes data out to a file per site.  I then have another program that reads each one of those site files and puts them into a SQLite DB.  I have another TCL proc that reads the DB and puts out a daily report.  I got the code from J.Cobane years ago and it works well.

Code to write it out to a data file:

Code:

HciStartup
set run_time [clock format [clock seconds] -format “%m-%d-%Y-%H:%M:%S”]
set Date [clock format [clock seconds] -format “%Y%m%d”]
set arg_values $argv
if { $argc < 2 } {
        puts "Usage is: get_stats_data SYSTEM SITE [EMAIL]"
        return
}
set system [lindex $arg_values 0]
set sitename [lindex $arg_values 1]
set email [lindex $arg_values 2]
set outfilename [append outfilename "/info/stats/" $system "_" $sitename "_statsfile_" $Date ".csv"]
set subject "get_stats_data for Sitename: $sitename at $run_time"
set debug 0
set outrec ""

if { [file exists $outfilename] } {
    set outfileID [open $outfilename a]
} else {
    set outfileID [open $outfilename w]
    append header "SYSTEM,SITENAME,THREADNAME,MSGSIN,MSGSOUT,BYTESIN,BYTESOUT"

    #append header "SITENAME,THREADNAME,SAMPLE_DAY,SAMPLE_DATE,SAMPLE_TIME,THREADSTART_DAY,THREADSTART_DATE,"
    #append header "THREADSTART_TIME,THREADSTOP_DAY,THREADSTOP_DATE,THREADSTOP_TIME,"
    #append header "PSTATUS,LASTREAD_DAY,LASTREAD_DATE,LASTREAD_TIME,"
    #append header "LASTWRITE_DAY,LASTWRITE_DATE,LASTWRITE_TIME,"
    #append header "LASTERR_DAY,LASTERR_DATE,LASTERR_TIME,LASTERRTEXT,"
    #append header "ERRORCNT,MSGSIN,MSGSOUT,BYTESIN,BYTESOUT,OBDATAQD,"
    #append header "IBLATENCY,OBLATENCY,TOTLATENCY,IBPRESMSQD,IBPOSTSMSQD,OBPRESMSQD,OBREPLYQD"
    puts $outfileID $header
}

# Attach to the Shared Memory Region
msiAttach
set threadlist [msiTocEntry]
set threadcount [llength $threadlist]

foreach threadname $threadlist {
   if { [string is digit [string range $threadname 0 0]] } {
               puts "$threadname skipped"
               continue    ;# This code added to skip threadnames that start with a digit
   }
       if { [string equal $threadname ""] } {
               continue
       }
   set threadkeys [msiGetStatSample $threadname]
   if {![string equal $threadkeys ""]} {
           set LASTEXTRACT [clock format [keylget threadkeys LASTEXTRACT] -format "%a,%m-%d-%Y,%H:%M:%S"]
           set THREADSTART [clock format [keylget threadkeys START] -format "%a,%m-%d-%Y,%H:%M:%S"]
           set THREADSTOP  [clock format [keylget threadkeys STOP] -format "%a,%m-%d-%Y,%H:%M:%S"]
           set PSTATUS     [keylget threadkeys PSTATUS]
           set PLASTREAD   [clock format [keylget threadkeys PLASTREAD] -format "%a,%m-%d-%Y,%H:%M:%S"]
           set PLASTWRITE  [clock format [keylget threadkeys PLASTWRITE] -format "%a,%m-%d-%Y,%H:%M:%S"]
           set PLASTERROR  [clock format [keylget threadkeys PLASTERROR] -format "%a,%m-%d-%Y,%H:%M:%S"]
           set PLASTERRTEXT [keylget threadkeys PLASTERRTEXT]
           set ERRORCNT    [keylget threadkeys ERRORCNT]
           set MSGSIN      [keylget threadkeys MSGSIN]
           set MSGSOUT     [keylget threadkeys MSGSOUT]
           set BYTESIN     [keylget threadkeys BYTESIN]
           set BYTESOUT    [keylget threadkeys BYTESOUT]
           set OBDATAQD    [keylget threadkeys OBDATAQD]
           set IBLATENCY   [keylget threadkeys IBLATENCY]
           set OBLATENCY   [keylget threadkeys OBLATENCY]
           set TOTLATENCY  [keylget threadkeys TOTLATENCY]
           set IBPRESMSQD  [keylget threadkeys IBPRESMSQD]
           set IBPOSTSMSQD  [keylget threadkeys IBPOSTSMSQD]
           set OBPRESMSQD  [keylget threadkeys OBPRESMSQD]
           set OBREPLYQD  [keylget threadkeys OBREPLYQD]
           set statsrec ""

           append statsrec "$system,$sitename,$threadname,$MSGSIN,$MSGSOUT,$BYTESIN,$BYTESOUT"

           #append statsrec "$sitename,$threadname,$LASTEXTRACT,$THREADSTART,$THREADSTOP,$PSTATUS,$PLASTREAD,"
           #append statsrec "$PLASTWRITE,$PLASTERROR,$PLASTERRTEXT,$ERRORCNT,$MSGSIN,$MSGSOUT,$BYTESIN,$BYTESOUT,$OBDATAQD,"
           #append statsrec "$IBLATENCY,$OBLATENCY,$TOTLATENCY,$IBPRESMSQD,$IBPOSTSMSQD,$OBPRESMSQD,$OBREPLYQD"

           puts $outfileID $statsrec

   }
}
close $outfileID

Code to read the csv files and write to SQLite DB:

Code:

package require sqlite
package require csv

sqlite DBCMD “/hci/cloverleaf/cis6.1/integrator/databases/engine_stats.db”
DBCMD timeout 6000

   HciStartup
   set run_time [clock format [clock seconds] -format “%m-%d-%Y-%H:%M:%S”]
   set Date [clock format [clock seconds] -format “%Y%m%d”]
   set arg_values $argv
   if { $argc 0 } {

       foreach file $locallist {
           if { [file isdirectory $file] } { continue }
           puts “file ==> $file”
           set FileId [open $file]
           set lines [split [read $FileId] “n”]
           close $FileId
           foreach line $lines {

               lassign [::csv::split $line $sepChar] system sitename threadname MSGSIN MSGSOUT BYTESIN BYTESOUT
               if { [string equal $system “SYSTEM”] } { continue }
               if { [string equal $system “”] } { continue }
               set sqlcmd “insert into engine_stats values(’$UserDate’,’$system’,’$sitename’,’$threadname’,$MSGSIN,$MSGSOUT,$BYTESIN,$BYTESOUT)”
               puts “sqlcmd ==> $sqlcmd”
               DBCMD eval $sqlcmd
           }
       }
   }

exit

Let me know if you have questions.

Rob

[Greg Tataryn]

Since the message and byte counts are a running total, if you need to get an hour over hour count, how are you going about that? Or are you just interested in daily/weekly totals and not hourly?

[Varun Sinha]

Hi Rob,

 I know it’s very old post, but I can see that you have shared

code to write it out to a data file

Code to read the csv files and write to SQLite DB

but I do not see the TCL proc that reads the DB and puts out a daily report.

Can you please share the TCL proc that reads the DB and puts out a daily report?

I am new to cloverleaf TCL. Appreciate your help in advance.

Regards,

Varun

[Rob Abbott]

Hi Varun, please note that in 6.2.0 the option to store statistics in a database is built into the core application.   You may want to leverage this rather than building something custom.

Rob Abbott
Cloverleaf Emeritus

[Dustin Sayes]

I’m very interested in learning about the statistic database.
Where do we configure which DB will be used for the statistic db?

[Author]

Replies