slow getting out of state 7

[Kevin Scantlan]

We have a puzzling situation here.  We have a number of threads that if we get an increase in volume of messages, will get a buildup of messages that queueing in the post xlate queue (state 7).  However, others with approximately the same volume do not.  Here’s an example:

Inbound thread A has 2 destination routes to what I’ll call thread B and thead C, both outbound threads.  They have similar amounts of traffic and similar processing.  But if the volume of messages coming inbound to A increases, you’ll see the post-xlate queue to C increase while it does not to B.  I have placed a script in the post-xlate route and the pre-outbound TPS of C to measure times.  There are times with it’s almost instantaneous, but there are times when it increases as much as 30 seconds.  Is this an ICL problem?  But if it were one would think it would be an across the site problem, which it is not.  

Any experience out there with something similar or any suggestions?  Maybe someone from Healthvision can shed some light on how the ICL works.

Topic

[Jim Kosloskey]

Kevin,

Are all of the threads in the same process?

Jim Kosloskey

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Michael Hertel]

If I were a betting man, I would say this is related to your previous post.

I am running 5.4.1 (special rev before rev1) on AIX ML6.

Here’s something from a previous post of mine you can look at.

Also, have you used topas to identify the offening process?

Try ulimit -a and see what your settings are. Here are mine.

hci rs6kg% ulimit -a

time(seconds)        unlimited

file(blocks)         2097151

data(kbytes)         1572864

stack(kbytes)        262144

memory(kbytes)       262144

coredump(blocks)     2097151

nofiles(descriptors) 10000

Here’s the info on maxuser processes from my Unix administrator:

root@rs6kg:/root>lsattr -El sys0 -a maxuproc

maxuproc 2048 Maximum number of PROCESSES allowed per user True

Hopefully your admin person can check this for you.

Here is info from our /etc/security/limits file:

default:

      fsize = 2097151

      core = 2097151

      cpu = -1

      data = 3145728

      rss = 524288

      stack = 524288

      nofiles = 10000

Lastly, check to see what mode you are running in.

64 bit or 32 bit

The box, the unix o/s and Cloverleaf need to be running in the same mode. I don’t know if 5.4.1 supports 32 bit mode anymore.

Hope this helps…

[Michael Hertel]

Another question, is FTP or SNA involved?

[Kevin Scantlan]

Jim,

No, the threads are in different processes.  In fact, both the threads A, B, and C are each in processes by themselves.

Michael,

Neither FTP nor SNA are involved.  I will check out the settings you gave me.

[Kevin Scantlan]

Here’s what our sys admin had to say:

Kevin,  All of our settings are the same or higher.  The machine is running the 64 bit kernel.

its-pluto# ulimit -a

time(seconds) unlimited

file(blocks) unlimited

data(kbytes) 1572864

stack(kbytes) 262144

memory(kbytes) 393216

coredump(blocks) 2097151

nofiles(descriptors) 10240

its-pluto# lsattr -El sys0 -a maxuproc

maxuproc 2048 Maximum number of PROCESSES allowed per user True

[Michael Hertel]

Did you have the same issue putting all the threads in one process?

[Kevin Scantlan]

We did not attempt to place all the threads in the same process.  Thread A and Thread B are in different processes, but have no problem when going from A to B.  Yet we have a problem with going from Thread A  to Thread C, which are also in different processes.  We did not want to overload one proces with it having both the inbound and the outbound threads.

[Michael Hertel]

Try putting them in one process.

My understanding is that there is additional overhead for going cross process. Cloverleaf works in a round robin fashion and is single threaded where it can only work on one thing at a time. Unless you have throttling involved, it will try to finish the first task before moving to the second. The since thread A delivers to thread B first and then to thread C, it may go A/B, A/C, A/B, A/B, A/B, A/C, etc. Remember, this is my opinon and theory.

Further, my opinion is that by putting them all in one process, the routing task will be smoother and more even since there is no cross processing involved.

Try it and see what happens. If it doesn’t work, put them back in separate processes.

[Jim Kosloskey]

Kevin,

I agree with Michael.

We avoid cross process here.

Sometimes for other reasons you need to have the engaged threads in different processes.

You can still avoid cross process in the same site by sending the messages from the inbound thread to an intermediate client thread which is tcp/ip or pdl tcp localhost. in the other process you would have a server thread same protocol as the intermediate client, same host (localhost) same port.

Now you have sent messages cross process without using the internal Cloverleaf(R) cross process handling.

Depending on the situation, this may be an indication it is time to embrace a multi-site architecture.

Of course on one of your other posts you indicated some system consumption issues. These could be related and it is also possible you might see some of the system consumption relieved after bypassing cross process. Then again, the system resource issue might simply be exacerbating the cross process situation.

Any way we have never regretted trying to eliminate cross process message delivery.

Jim Kosloskey

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies