Reset and Collect Statistics into a DB – TCL – hcimsiutil

Homepage Clovertech Forums Read Only Archives Cloverleaf Cloverleaf Reset and Collect Statistics into a DB – TCL – hcimsiutil

  • Creator
    Topic
  • #54919
    Erik Mueller
    Participant

    Hi,

    I’m wondering if anybody has written something similar to this already that is willing to share it with me.

    I’m wanting to collect the statistics for the threads on a regular basis and then reset them e.g. once a week, I collect the stats and then reset them.

    Has anybody written a TCL that will extract the statistics for a thread and store them in a SQLite DB (or a text file)?

    I’m not interested in all the statistics, just the Msgs In, Msgs Out, Xlated and Failed.

    Anything out there would be appreciated,

    Thanks,

    Erik

Viewing 6 reply threads
  • Author
    Replies
    • #83452
      Jim Kosloskey
      Participant

      I have an old proc that dumps the entire msi stats to a delimited file whiich could then be imported to any tool allowing importation of a delimited file (DB included).

      Email me and I will send it – you can do with it what you want.

      email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

    • #83453
      Elisha Gould
      Participant

      I’d actually be interested in this as well. I was just thinking about this the other day to help with statistics when users ask about volumes throughout the day, and also to help detect potential issues on an interface.

    • #83454
      Rob Lindsey
      Participant

      I have something that writes data out to a file per site.  I then have another program that reads each one of those site files and puts them into a SQLite DB.  I have another TCL proc that reads the DB and puts out a daily report.  I got the code from J.Cobane years ago and it works well.

      Code to write it out to a data file:

      Code:


      HciStartup
      set run_time [clock format [clock seconds] -format “%m-%d-%Y-%H:%M:%S”]
      set Date [clock format [clock seconds] -format “%Y%m%d”]
      set arg_values $argv
      if { $argc < 2 } {
              puts "Usage is: get_stats_data SYSTEM SITE [EMAIL]"
              return
      }
      set system [lindex $arg_values 0]
      set sitename [lindex $arg_values 1]
      set email [lindex $arg_values 2]
      set outfilename [append outfilename "/info/stats/" $system "_" $sitename "_statsfile_" $Date ".csv"]
      set subject "get_stats_data for Sitename: $sitename at $run_time"
      set debug 0
      set outrec ""

      if { [file exists $outfilename] } {
          set outfileID [open $outfilename a]
      } else {
          set outfileID [open $outfilename w]
          append header "SYSTEM,SITENAME,THREADNAME,MSGSIN,MSGSOUT,BYTESIN,BYTESOUT"

          #append header "SITENAME,THREADNAME,SAMPLE_DAY,SAMPLE_DATE,SAMPLE_TIME,THREADSTART_DAY,THREADSTART_DATE,"
          #append header "THREADSTART_TIME,THREADSTOP_DAY,THREADSTOP_DATE,THREADSTOP_TIME,"
          #append header "PSTATUS,LASTREAD_DAY,LASTREAD_DATE,LASTREAD_TIME,"
          #append header "LASTWRITE_DAY,LASTWRITE_DATE,LASTWRITE_TIME,"
          #append header "LASTERR_DAY,LASTERR_DATE,LASTERR_TIME,LASTERRTEXT,"
          #append header "ERRORCNT,MSGSIN,MSGSOUT,BYTESIN,BYTESOUT,OBDATAQD,"
          #append header "IBLATENCY,OBLATENCY,TOTLATENCY,IBPRESMSQD,IBPOSTSMSQD,OBPRESMSQD,OBREPLYQD"
          puts $outfileID $header
      }

      # Attach to the Shared Memory Region
      msiAttach
      set threadlist [msiTocEntry]
      set threadcount [llength $threadlist]

      foreach threadname $threadlist {
         if { [string is digit [string range $threadname 0 0]] } {
                     puts "$threadname skipped"
                     continue    ;# This code added to skip threadnames that start with a digit
         }
             if { [string equal $threadname ""] } {
                     continue
             }
         set threadkeys [msiGetStatSample $threadname]
         if {![string equal $threadkeys ""]} {
                 set LASTEXTRACT [clock format [keylget threadkeys LASTEXTRACT] -format "%a,%m-%d-%Y,%H:%M:%S"]
                 set THREADSTART [clock format [keylget threadkeys START] -format "%a,%m-%d-%Y,%H:%M:%S"]
                 set THREADSTOP  [clock format [keylget threadkeys STOP] -format "%a,%m-%d-%Y,%H:%M:%S"]
                 set PSTATUS     [keylget threadkeys PSTATUS]
                 set PLASTREAD   [clock format [keylget threadkeys PLASTREAD] -format "%a,%m-%d-%Y,%H:%M:%S"]
                 set PLASTWRITE  [clock format [keylget threadkeys PLASTWRITE] -format "%a,%m-%d-%Y,%H:%M:%S"]
                 set PLASTERROR  [clock format [keylget threadkeys PLASTERROR] -format "%a,%m-%d-%Y,%H:%M:%S"]
                 set PLASTERRTEXT [keylget threadkeys PLASTERRTEXT]
                 set ERRORCNT    [keylget threadkeys ERRORCNT]
                 set MSGSIN      [keylget threadkeys MSGSIN]
                 set MSGSOUT     [keylget threadkeys MSGSOUT]
                 set BYTESIN     [keylget threadkeys BYTESIN]
                 set BYTESOUT    [keylget threadkeys BYTESOUT]
                 set OBDATAQD    [keylget threadkeys OBDATAQD]
                 set IBLATENCY   [keylget threadkeys IBLATENCY]
                 set OBLATENCY   [keylget threadkeys OBLATENCY]
                 set TOTLATENCY  [keylget threadkeys TOTLATENCY]
                 set IBPRESMSQD  [keylget threadkeys IBPRESMSQD]
                 set IBPOSTSMSQD  [keylget threadkeys IBPOSTSMSQD]
                 set OBPRESMSQD  [keylget threadkeys OBPRESMSQD]
                 set OBREPLYQD  [keylget threadkeys OBREPLYQD]
                 set statsrec ""

                 append statsrec "$system,$sitename,$threadname,$MSGSIN,$MSGSOUT,$BYTESIN,$BYTESOUT"

                 #append statsrec "$sitename,$threadname,$LASTEXTRACT,$THREADSTART,$THREADSTOP,$PSTATUS,$PLASTREAD,"
                 #append statsrec "$PLASTWRITE,$PLASTERROR,$PLASTERRTEXT,$ERRORCNT,$MSGSIN,$MSGSOUT,$BYTESIN,$BYTESOUT,$OBDATAQD,"
                 #append statsrec "$IBLATENCY,$OBLATENCY,$TOTLATENCY,$IBPRESMSQD,$IBPOSTSMSQD,$OBPRESMSQD,$OBREPLYQD"

                 puts $outfileID $statsrec

         }
      }
      close $outfileID

      Code to read the csv files and write to SQLite DB:

      Code:


      package require sqlite
      package require csv

      sqlite DBCMD “/hci/cloverleaf/cis6.1/integrator/databases/engine_stats.db”
      DBCMD timeout 6000

         HciStartup
         set run_time [clock format [clock seconds] -format “%m-%d-%Y-%H:%M:%S”]
         set Date [clock format [clock seconds] -format “%Y%m%d”]
         set arg_values $argv
         if { $argc 0 } {

             foreach file $locallist {
                 if { [file isdirectory $file] } { continue }
                 puts “file ==> $file”
                 set FileId [open $file]
                 set lines [split [read $FileId] “n”]
                 close $FileId
                 foreach line $lines {

                     lassign [::csv::split $line $sepChar] system sitename threadname MSGSIN MSGSOUT BYTESIN BYTESOUT
                     if { [string equal $system “SYSTEM”] } { continue }
                     if { [string equal $system “”] } { continue }
                     set sqlcmd “insert into engine_stats values(’$UserDate’,’$system’,’$sitename’,’$threadname’,$MSGSIN,$MSGSOUT,$BYTESIN,$BYTESOUT)”
                     puts “sqlcmd ==> $sqlcmd”
                     DBCMD eval $sqlcmd
                 }
             }
         }

      exit

      Let me know if you have questions.

      Rob

    • #83455
      Greg Tataryn
      Participant

      Since the message and byte counts are a running total, if you need to get an hour over hour count, how are you going about that? Or are you just interested in daily/weekly totals and not hourly?

    • #83456
      Varun Sinha
      Participant

      Hi Rob,

       I know it’s very old post, but I can see that you have shared

      code to write it out to a data file

      Code to read the csv files and write to SQLite DB

      but I do not see the TCL proc that reads the DB and puts out a daily report.

      Can you please share the TCL proc that reads the DB and puts out a daily report?

      I am new to cloverleaf TCL. Appreciate your help in advance.

      Regards,

      Varun

    • #83457
      Rob Abbott
      Keymaster

      Hi Varun, please note that in 6.2.0 the option to store statistics in a database is built into the core application.   You may want to leverage this rather than building something custom.

      Rob Abbott
      Cloverleaf Emeritus

    • #109096
      Dustin Sayes
      Participant

      I’m very interested in learning about the statistic database.
      Where do we configure which DB will be used for the statistic db?

Viewing 6 reply threads
  • The forum ‘Cloverleaf’ is closed to new topics and replies.

Forum Statistics

Registered Users
5,105
Forums
28
Topics
9,278
Replies
34,382
Topic Tags
281