Load flat files without breaking recovery DB

Clovertech Forums Cloverleaf Load flat files without breaking recovery DB

  • Creator
    Topic
  • #119637
    Yutaka
    Participant

      We’re trying to load 94 HL7 flat files ~15-30MB in size.  Currently if I load ~120MB worth of files into the inbound thread filepath, it seems like all the messages have to get indexed first, then gets loaded into recovery database.  So while it takes some time to output messages on the outbound thread, recovery database gets overloaded and crashes the site.  Is there any setting in Cloverleaf where I can dump all the files into the filepath, then Cloverleaf will wait to move onto the next file while one file is loaded in recovery database and is in the process of being processed?

    Viewing 1 reply thread
    • Author
      Replies
      • #119638
        Charlie Bursell
        Participant

          Why load all at one time?  Use Fileset/Local and you can meter the number of records read and how often

        • #119639
          Jeff Dinsmore
          Participant

            When resending a large number of messages, I’ll  frequently just run a script from a Tcl shell.

            I’ll loop through messages one at a time – write each message to a temporary file and queue messages with hcicmd using a command something like this:

             

            if { [catch {exec hcicmd -p $process -c “$thread resend ib_pre_tps data 5120 /tmp/requeueFile.txt nl”} err] } { puts “ERROR: $err” }

             

            You need to setsite to the proper site prior to queueing the messages.

            To throttle the message flow, I’ll insert a sleep for a variable number of milliseconds between each message.  The number of msec is read from a file.

            I keep track of the last mod time of that file and compare it to current mod time of the file.  When the mod time changes, I update the sleep msec between messages.  Something like this:

             

            if { [file exists $sleepFilePath] } {

            set sleepFileMtime [file mtime $sleepFilePath]

            if { $sleepFileMtime > $lastSleepMtime } {

            set sf [open $sleepFilePath r]
            set tmpSleepMsec [string trim [read $sf]]
            close $sf
            if { [string is digit $tmpSleepMsec] } {
            puts “setting sleepMsec to $tmpSleepMsec”
            set sleepMsec $tmpSleepMsec
            }
            set lastSleepMtime $sleepFileMtime
            }

            } else {
            set sf [open $sleepFilePath w]
            puts $sf $sleepMsec
            close $sf
            set lastSleepMtime [file mtime $sleepFilePath]
            }

             

            This allows you to fine-tune the resend rate on-the-fly.

            Start off conservative with a second (1000 msec) between messages.  Monitor recovery DB queue depth and adjust the sleep time accordingly.  Reduce it to send messages faster. Increase it if the queue grows.

            As soon as you write a new value and save the file, the script will see the mod time has changed, read the new value from the file and adjust the sleep msec.

            After a short tuning time, you’ll arrive at a number that sends the messages as quickly as possible, but doesn’t result in a recovery DB that grows too large.

             

             

             

            Jeff Dinsmore
            Chesapeake Regional Healthcare

        Viewing 1 reply thread
        • You must be logged in to reply to this topic.