Process consuming high amount of memory

[Allerson Pereira]

Hi friends, I have a situation here.

I have a process with one single thread. This thread run over time reading registers from a database and writing into another. there is no message traffic here.

I just open a connection, generate a list of results, close connection, open a new connection into destiny DB, insert all values of my list and close connection.

Memory consumption of this process is increasing about 1mb per minute. It is to much because I have to cycling every 6-7 hours.

The deal is, how can I identify what is going wrong? how to clean up used memory?

At the moment I have no idea so any help will be appreciate.

TIA

Allerson

Topic

[Russ Ross]

Sounds like a possible memory leak in that process.

A quick way to check if that is the case is to stop the process then look near the bottom of the process log file to see if it shows leaked handles being cleaned up when the process is shutdown.

If you determine that is happening then that will narrow your search down to some of the typical ways an integration can be built with leaked handles.

If your integration is making a modified copy of the incoming message and sending out the modified copied without cleaning up the incoming message that was copied, that is a common casue of memory leaks.

Look for msg or grm commands to see if any of these are being called and then see if they are being cleaned up after calling them.

Russ Ross
RussRoss318@gmail.com

[Allerson Pereira]

Hi friends, here an update.

After some experiments, we discovered an “weird” behavior from Oracle ODBC. We noticed after observe oracle log files that some connections (almost all of them) were not closed, remaining active in memory. I spent a couple of hours trying to fix this but in the end I just replaced that driver for an of version 11 and now everything looks good.

Honestly I still don’t know what was going wrong and how to fix it so I can’t share anything else with you.

Thank you

[Arslan Khan]

Hi Russ,

We have many processes consuming more than normal amount of memory (some time > 100MB).

When I check the process logs, I do see this below leaked handles:

[cmd :cmd :INFO/0:ahs_rslt_out_cmd:05/23/2013 08:47:29] Receiving a command

[cmd :cmd :INFO/0:ahs_rslt_out_cmd:05/23/2013 08:47:29] Command client went away.  Closing connection.

[prod:prod:INFO/0:ahs_rslt_out_xlate:05/23/2013 08:47:29] Checking for leaked handles in the Xlate interpreter…

Handle     Allocated by

======     ============

datum0     trxidbygrmaddr message0 grm0 {0(0).MSH(0).#3(0).[0]}

datum1     trxidbygrmaddr message0 grm0 {0(0).MSH(0).#3(0).[0]}

datum2     trxidbygrmaddr message0 grm0 {0(0).MSH(0).#3(0).[0]}

datum3     trxidbygrmaddr message0 grm0 {0(0).MSH(0).#3(0).[0]}

datum4     trxidbygrmaddr message0 grm0 {0(0).MSH(0).#3(0).[0]}

datum5     trxidbygrmaddr message0 grm0 {0(0).MSH(0).#3(0).[0]}

For some processes, there are 1000

[Russ Ross]

I would start by looking at the TCL procs being called by any process that shows the leaked handles in the log file and see what is being allocated but not cleaned up.

The most common causes are making a copy of an incoming message to make an outgoing message without cleaning up the incoming message or the copy, whichever the case may be.

Here is a best practice I do when working with datums

Code:

   #————————————————–
   # save the current datum list
   # that will need to be reset when this proc is done
   #————————————————–

   set datum_list [datlist]

   .
   .
   .

   #——————————————————————-
   # reset the datum list back to what is was when this proc was called
   #——————————————————————-

   hcidatlistreset $datum_list

If creating a grm handle be sure to clean it up before processing the next message something like this example:

Code:

set ghd [grmcreate -msg $mh hl7 2.3.1 blood_bank_progesa ORM_O01]
.
.
.
if {![cequal $ghd “”]} {
   grmdestroy $ghd
}  

The thing to keep in mind if any action is taken to create something be sure an action is taken to clean it up before processing the next message in the pipe.

We are human and overlook this but the process log files help bring it to our attention as well as monitoring processes to see if any constantly grow in RAM.

Sounds like you are gaining insight to narrow the focus and find some if not most of them and what might be causing them.

Russ Ross
RussRoss318@gmail.com

[Arslan Khan]

Thanks for the feed back Russ.

We were able to pin point the cause of these memory leaks. We are working on a fix now and hoping that this will eliminate this high memory/cpu utilization problem.

[Bob Richardson]

Greetings,

Though we rarely use the GRM method to get fields and test in our

TCL procedures we do have a few.

Here is a laundry list of items to check for cleanup.  The “datPatType”

was a variable that the code created to test patient type.

Hope it helps.

#

# Clean up temporary storage.

#

           grmdestroy $grmId

           datdestroy $datId

           datdestroy $datPatType

           hcidatlistreset $datList

[Arslan Khan]

I was thinking that it was some tclproc which was causing this issue. Actually, what I am noticing now is that it is the use of

[Bob Richardson]

This is new territory as of 5.8.5 for sure and I would strongly recommend that you log a case for INFOR support.  We were curious about using this new trxid method ourselves.

Good catch!   They are working on a Rev 7 for 5.8 and there is a Rev 1 for 6.0 now too.

[Russ Ross]

Bob:

We are in the mdist of upgrading to cloverleaf 6.0 and were wondering if there would be a rev 1 release before we got done.

Can you point me where to locate the cloverleaf 6.0 rev 1 download?

I don’t see it on the same download page I got the base 6.0 install from.

Russ Ross
RussRoss318@gmail.com

[Bob Richardson]

Russ,

I am unable to find this patch myself at the INFOR download center.

How about contacting INFOR support and confirm that it really is in development.

I may be in error and will confess it if necessary to this Forum in

the interests of telling the truth here.

Thank you.

[Marc Pleijers]

Hi,

According to infor

   * Revision patch 6.0.1 for Cloverleaf 6.0

   * Revision patch 5.8.7 for Cloverleaf 5.8

are both expected to become available in July.

Beta test for 6.0.1 has just been started.

regards,

Marc

Marc Pleijers
Senior Integration Consultant
Enovation BV
The Netherlands

[Author]

Replies