to SMAT or not to SMAT…

Interesting, you have an I/O issue and you think it is SMAT and not your home-grown tracing activity that is causing the issue?

It would seem to me you could get your trace file (or the information it represents) after the fact by processing the SMAT files in a batch environment rather than taking the hit with each interaction.

I wonder if it is really necessary to have the trace file be real time.

Hi Troy,

possible gotchas to not keeping the outbound SMATs

If any of your XLTs or TPS procs depend upon tables that change, you may have some latency issues. Replaying an old set of inbound data through a newer route/TCL/table/Xlate configuration might not exactly replicate what did in fact happen.

There’s also the hassle of re-playing messages to only one outbound thread if you have a one to many route from he inbound.

The good news is that I believe your hardware is adequate to handle your load because your are practicaly describing what we are running on with a similar number of threads with about a 13 million a day message flow.

If the SAN which is probably out of your control is the cause of the disk I/O bottle neck then a new server using the same SAN group might not be a significant improvement.

This was a serious concern I had and ran into when we were first forced to move from local serial disk to SAN.

I spent the next 3 years suffering while the SAN group finally got it runniong fast enough and reliably.

We have a very similar setup of the hardware, OS, cloverleaf, thread count.

We do have more site granualaity but have SMAT on just about everything and keep up nicely with about a 13 milliong message flow per day with a comfortable amount of horse power to spare.

My previous platform did force me to get very creative to stay afloat and I quickly discovered disk I/O to be the single biggest bottleneck which typically is always true of any platform or application.

Since I’ve witnessed first hand the capacity of your hardware is likely adequate, I like Jim suspect your home grown tracing might be suspect and would take a look at it to see if it is opening and closing the file with every message.

You are welcome to call me if you want to leverage my experience since I’ve already made the journey you are talking about embarking on.

One quick thought that pays big returns towards reducing wasted use of resources, is to filter unecessary message as far upstream as possible.

In fact, we don’t allow the use of static raw route and require a route for each message type to help in this effort.

Also, make sure EO config logging is not set to all except during times of troubleshooting.

Okay enough or I will get going and find it hard to stop with so many things that all add up.

I’m curious with such similar hardware what is the most queue dethp of messages you’ve seen without the process panicing?

Our old server tanked at about 8,000 queue depth but on our current server we hit over 150,000 and kept working fine, which is a much nicer cushion, but leaves me wondering what my upper limit might be.

It dawned on me that even though I’m running AIX with 8 processors and a healthy amount of RAM like you, that doesn’t mean you are running on P570s like I am.

In my previous post for some reason I had draw a conclusion that you too are running on P570s but that might not be true.

So when I say your machine might be big enough to handle your load, I was talking about P570s or bigger.

Hi Troy,

At my previous job,  working with AIX and eGate, I found that I/O issues were usually related to server hardware- on many occassions it was the controller card. Does the error report (errpt) on AIX show any h/w related errors ? You could also gather statistics from “iostat” to look for SAN performance issues.

Thanks.