Clovertech

Memory Leak in CL 6.1.1?

› Clovertech Forums › Read Only Archives › Cloverleaf › Cloverleaf › Memory Leak in CL 6.1.1?

  • Creator
    Topic
  • September 7, 2016 at 5:35 pm #55189
    Jeff Dinsmore
    Participant

      I have what appears to be a memory leak in 6.1.1

      Periodically over the last several days – currently about once per day – I’m getting the following error in several of my processes – then they shut down:

      Quote:

      WARNING: engine terminating due to swap space shortage

      All processes have a default static route with a Generate call to hcitpsmsgkill, so there should be no leaked message handles.

      I was watching “top” and noticed several processes that were using relatively large amounts of swap, so I restated them.  

      That freed up some swap and hard RAM and will buy me some time, but my hard RAM free is headed steadily downhill and it will eventually start to eat up swap again, I’m sure.  

      90 minutes ago, I had 4GB of RAM available, now I’m below 2.5GB.

      Most of the processes using the largest amounts of memory were inbounds that are just raw routing (via pdl-tcpip/tcp_acknak.pdl) to other localhost threads where the actual message manipulations are being done.  The only Tcl code used by these inbound/distribution processes is a simple trxid script.

      The only significant change I’ve made over the last several days was to change one of my utility SQLite dtabases to use write-ahead logging.  I can’t imagine that’s the culprit, but that change is the only one that’s roughly coincident with the onset of this problem.

      So, to the question… Have any of you seen any problems with memory leaks in CL6.x?

      Any tips/tricks to minimize memory consumption?

      Just checking here before I open a case with Infor support.

      Thanks!

      Jeff Dinsmore
      Chesapeake Regional Healthcare

    • Creator
      Topic
    Viewing 6 reply threads
    • Author
      Replies
      • September 7, 2016 at 5:54 pm #84511
        James Cobane
        Participant

          We have been running 6.1.1 since August of 2015 and have not noticed this type of issue.

        • September 7, 2016 at 6:00 pm #84512
          Jeff Dinsmore
          Participant

            The trxid script is here

            Code:

            ######################################################################
            # Name: trxid_hl7_system.tcl
            #
            proc trxid_hl7_system { msgId } {
            # Retrieve the Message Type from the message
            set msg [msgget $msgId]
            set fs [string index $msg 3]
            set sfs [string index $msg 4]

            set fieldList [split $msg $fs]

            set msgType [lindex [split [lindex $fieldList 8] $sfs] 0]
            set sendingApp [lindex $fieldList 2]

            return “$msgType_$sendingApp”
            }

            I haven’t shut down all of my processes looking for leaked message handles, but will do that to confirm.

            Thanks!

            Jeff Dinsmore
            Chesapeake Regional Healthcare

          • September 7, 2016 at 7:19 pm #84513
            James Cobane
            Participant

              The TRXID proc looks pretty benign, so I think you are good there.  You probably should look at the SQLite stuff you’re doing since you indicated in your own words “…that change is the only one that’s roughly coincident with the onset of this problem.” 🙂

              Good luck!

              Jim Cobane

              Henry Ford Health

            • September 7, 2016 at 7:20 pm #84514
              John Mercogliano
              Participant

                Jeff,

                  You didn’t mention what OS but there was an issue with AIX 6.1 TL9 SP1.  

                Here is the link.

                <a href="https://usspvlclovertch2.infor.com/viewtopic.php?t=6976&highlight=swap&#8221; class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?t=6976&highlight=swap

                John Mercogliano
                Sentara Healthcare
                Hampton Roads, VA

              • September 7, 2016 at 7:52 pm #84515
                Jeff Dinsmore
                Participant

                  We can stop looking.  I think I found the problem – and it’s sitting in my chair…

                  I stopped all of my processes and on the Very Last One, I found a script that was not killing messages as it should.  It was filtering some ADT triggers, but letting others fly with no disposition.

                  It’s been that way since at least March, but took until last weekend to bite me.

                  My free RAM is still falling a little, but may just be leveling off after restarting all processes.

                  Sorry for the false alarm and, as always, thanks for your quick responses.

                  Jeff Dinsmore
                  Chesapeake Regional Healthcare

                • November 18, 2016 at 8:08 pm #84516
                  Jeff Dinsmore
                  Participant

                    Following up…

                    Top is still showing that free memory on my Linux server is trending toward zero.

                    However, I found an interesting explanation of this today that suggests this is normal behavior and doesn’t indicate a exhaustion of memory at all.

                    Here’s a link for your reading enjoyment:

                    <a href="http://www.linuxatemyram.com/&#8221; class=”bbcode_url”>http://www.linuxatemyram.com/

                    Happy Thanksgiving to all!

                    Jeff Dinsmore
                    Chesapeake Regional Healthcare

                  • November 22, 2016 at 10:17 am #84517
                    Robert Kersemakers
                    Participant

                      Thanks for the info Jeff. When we started using Linux, we also saw with ‘top’ that all memory was in use. We had to be reassured by maintenance that this wasn’t the case. Your link explains it perfectly.

                      Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

                  • Author
                    Replies
                  Viewing 6 reply threads
                  • The forum ‘Cloverleaf’ is closed to new topics and replies.