Performance of using Branch vs a group of route details.

Do you have any idea what may have caused the sudden latency issue?

Was there a sudden increase in the volume of messages to be handled?

Were there any changes to the objects in the routes?

Were there any changes to the platform or operating environment?

Was more work (outside of the Branch routes) added to either the same or other processes in the site?

I am just curious as to what may have caused the sudden degradation.

Unless this was not sudden and simply has been getting progressively worse and just arrived at a pain threshold.

Thanks.

Probably the last point. We have been gredually increasing the threads and processes in the site. But this has been an ongoing thing and all of a sudden the complaints started coming in about getting ADT messages only after a long delay.

When overloading processes (or to a lesser degree sites), the effect is not gradual. Rather a plateau eventually gets reached and it is like falling off of a cliff.

Normal capacity and performance measurements such as CPU/Memory consumption or SIO saturation are insufficient in this model to give early warning.

Rather it is essential in my opinion to understand the peak arrival rate at each point of load transition and measure the latency through the engine watching carefully for degradation and especially for queue depth buildup.

So lets drill down some with what we know.

Did any of these branched routes cross processes? Was the increase in additional work also in the process(es) which housed the branches or other processes?

During the performance issue were any of the outbound threads or routes suffering from deep queues?

Is the cross process routing internal (in other words the route from an inbound thread in one process routes to another thread in a different process) or was cross process achieved by utilizing local host threads (IB routed to OB local host in one process then an IB thread local host in another process picks up the messages and delivers them to the final OB destination?

I have not used the branch so I am not sure internally where the consumption points might be. If it is possible to increase the EO when the branches are in use and study the engine activity then one could compare that to the non-branched solution to possibly see where the additional engine activity might be.

It is interesting that there were no deep queues (route thread or pending) during the slowdown. Is it possible the messages were not arriving from the source as quickly as they were created – in other words could there have been a queue on the source?

If you would like to email me maybe we could continue this discussion off line.

We’ve had issues where everything slows down considerably when the SAN is having issues.

In our case we need the io latency to be below 10ms. If it bumps up past this threshold, we start seeing slowdowns in throughput.

If you keep logs of the server activity, it’ll show higher than normal io wait if this occurs.