Paging space increase after applying 6.0.2 on AIX 6.1 TL7

[Peter Heggie]

Paging space went from an average of 5% to 65%. This is on our test server.

I issued  

hcicitectl -K

hcidbinit -A -C

hcimsiutil -Z

hcimsituil -R

hcisitectl -S

I also stopped the host server.

After doing that for all sites, the paging space fell to 12%.

It stayed at 12% all morning. we checked every hour. At 2pm it jumped to 69%. We doubled the paging space. We found one instance of the hcixlttest -i using high cpu and high memory. At the time an HRL was being tested. After we shutdown that testing tool and recycled the process being worked on, the paging space fell to 30%. It later went back up to 45% but without an obvious culprit.

When I run topas, I see that java hci is using 89.6% of paging space, but I’m not sure how to tell which java process is responsible. We have several database threads which use Java, I’m not sure what else uses java. We do not have jav upocs.

We are using the Direct Adapter but it is not currently running.

Any thoughts on how to narrow the search?

Thanks,

Pete

Peter Heggie
PeterHeggie@crouse.org

Topic

[Peter Heggie]

Since learned that the java process in question was the host server, although other high usage processes, but not all, have a db protocol. None of our processes have more than one db protocol thread.

In the morning I will try recycling the host server again.

Peter Heggie
PeterHeggie@crouse.org

[Bob Richardson]

Greetings,

I urge you to post an incident at INFOR’s website and provide your information.   The Java hostserver may have memory leaks that need to be plugged.  Or other processing inefficiencies.   INFOR can guide you on setting your hostsever in debug mode to gather more details too.

Our shop is closely watching these posts on the latest Integrator release as that is our next step up from our current 5.8.7.0 version.

Thanks.

[Peter Heggie]

thank you – I have opened an incident with Infor and we are digging into it, and I will relay the findings here.

Peter Heggie
PeterHeggie@crouse.org

[Peter Heggie]

It is very likely that using the testing tool on translates is causing the problem. The particular scenario is with X12 EDI (834) on the output and a complex HRL on the input. I don’t know if it is the input HRL or the output X12 that is the problem.

Every time we run that scenario, the paging space utilization goes up. The impact seems to be broad – our top ten paging space processes run in several different sites, so it is not concentrated in one site or one process. So what I am saying is that when we run the testing tool with that input and output, many processes, in many sites (on the same server & Cloverleaf release), will increase in utilization.

We recycle all the processes when we can and the utilization goes down.

Here is an important point – we had never run this scenario before we upgraded to 6.0.2! So this problem might exist in previous releases.

The input configuration is a top-level HRL having several child-level FRLs and one child-level HRL having grand-children FRLs.

Pete

Peter Heggie
PeterHeggie@crouse.org

[Jim Kosloskey]

Pete,

To nail down input or output, create 2 more Xlates. One with the HRL definition and maybe a FRL, then COPY a literal to one outbound field. Do a Xlate Test – if the problem occurs then it is the HRL. Do the same thing just using the X12.

Have you tried to define this in a running engine to see if it the same thing happens outside the Testing Tool?

I know we recently saw a increase in Virtual Storage usage – turned out to be an AIX issue – a patch had a bug in it.

I think Russ posted something about it here so try to find that (I will see if I can locate it as well).

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 60 years IT – old fart.

[Russ Ross]

Here is the post Jim is talking about:

<a href="https://usspvlclovertch2.infor.com/viewtopic.php?p=31359#31359&#8243; class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?p=31359#31359

which in our case the version of AIX we upgraded to has a known problem with not releasing paging space after the application is done with it.

Russ Ross
RussRoss318@gmail.com

[Brenda Carpenter]

Hi Jim,

You and I corresponded a bit a couple months ago regarding my task of translating Lawson 834 FF to X12   😀   I will try creating the separate xlates and run isolated tests on those as well to find where the paging space increase may be occurring, but I have executed the HRL testing tool alone with this HRL it caused the paging space to go up quite a bit.

I do have a test thread built using this xlate and have executed it

a few times (with no successful output yet) but this has not

increased the paging space usage.

Thanks..Brenda

Crouse Hospital, NY

[Jim Kosloskey]

Brenda,

Does the paging space reduce  once you close the tester?

It is interesting the paging space is not affected when in a running engine.

I would expect the tester to potentially require more virtual while it is executing but I would also expect the pages to be released once the tester is closed.

email: jim.kosloskey@jim-kosloskey.com 30+ years Cloverleaf, 60 years IT – old fart.

[Brenda Carpenter]

Hi Jim,

No it doesn’t free the space, unfortunately.  We are manually recycling  the test sites to free the space back up.    

Yesterday morning, “normal” pagings space used:  15%, I executed one test in the Xlate test tool and paging space went to 32% – it did return an error (after some time) in my translator that I corrected, but I have not yet re-run it.

This morning I recycled the four test regions and the space went down to 17%.  This afternoon I executed one test HRL test in test tool and the space went back up, now to 38%.   I again recycled the four test regions, but this time it did not free any of the space.

I agree it is strange that running the engine thread does not impact it, but then again, I have yet to get output from this process..

Brenda

[Brenda Carpenter]

Good news, this morning I modified the secondary HRL to change the repeats section for two of the segments that repeat and I am no longer getting the pagination space issues using the xlate test tool.

format:

HRL – Main HRL being tested

 FRL – header

 FRL

 FRL

 FRL

 HRL – This HRL contains 12 FRLs and can repeat up to 9999 times

   FRL

   FRL  – repeats, had 99 here, changed to 3

   FRL  – repeats, had 99 here, changed to Repeat while…. since the

            number of these varies per loop

   9 more FRLs none repeat

FRL – trailer

One thing also to note when using the Flat file from Lawson for input, is that the translate process was not recognizing the end of line, I had to code the carriage return one byte at the end of each of the FRL layouts.

I still have a lot of tweaking to go with this layout, but at least I am able to run the test tool!!!

Thanks! Brenda

[Author]

Replies