How to prioritize the Command Threads

[Ivan Ng]

I have two threads setup in two different processes.  The inbound thread reads messages using java upoc.  The outbound thread uses TCP/IP to send messages.  The inbound thread passes the messages it read to the outbound thread in raw.

What happens is that the inbound thread keeps reading the messages and all those messages are being queued in the route (state 7).  One or two messages may get passed to the outbound thread from time to time but as long as the inbound thread is still reading messages, most messages are being queued in the route.

It seem that the command thread does not have time to move the messages from state 7 to the outbound thread.  Is that any way to fix this?  I have 100k messages to transfer and with this problem, the recovery db will be jammed.

The bottom line is that the inbound thread has to keep reading messages as long as there is no messages in the route.  As a result, setting a larger  value for the upoc’s read interval is not acceptable.

Topic

[Mike Grieger]

I believe this is an example of where you would use Translation Throttling (configured in the NetConfig Process Configuration).  There you can set a max number of messages to process before giving time to other processes (see the online Help guide for more into on this).

[Ivan Ng]

That’s what I have just figured out as well.

However, it seem that I did not pay attention to the true problem.  The true problem is that the outbound thread is slower than the inbound thread.  As such, no matter what I do there will be a large number of messages being queued in the recovery db unless I choose a very long read interval for the inbound thread.  In any case, it would be very difficult to optimize for the maximum speed.

It would be ideal if cloverleaf can “wait” on the inbound thread whenever it has nothing to do.

[Ivan Ng]

By the way, it would also be nice if someone can shed some light on how to further prioritize the command thread over the translation.

Even with throttling set to (1, 1, 50) which means that only 1 message is translated everytime, there are still lots of messages left in the post-transaltion queue.  Is there any way to force the command thread to move those messages to the outbound thread prior to translating more messages?  The outbound thread is in another process, so having those messages moved to the outbound thread first will speed things up.

I am sure that the outbound thread is not saturated yet since the message out count increases faster when it is being feed with messages continuously than when most messages are stuck in the post-translation queue.  By the way, it never gets any pending messages.

[Russ Ross]

One simple way to speed up your integration is to put the outbound thread in the same process as the inbound thread.

Cross processes routing is slow and in my opinion should be avoided.

The only time I use cross process routing is when I have 2 inbound threads in differnet processes feeding one outbound thread.

I’m not sure if my next thought is a desirable solution but you could wait for the ACK from the outbound thread and route that ACK back to the inbound thread before the inbound thread sends an ACK which means the integration will only run as fast as the weakest link (the outbound thread).

Russ Ross
RussRoss318@gmail.com

[Ivan Ng]

I don’t know that cross-process routing is slow.  Thanks for pointing that out.  But since my outbound is java upoc and is cpu intensive as well, so I am not so sure if it is beneficial to put it in the same process as the inbound thread.

As for the ACK, I think that’s a good idea, but it doesn’t work in my case because the inbound messages is sent to two outbound threads and some of the messages will get dropped during translation.

[Russ Ross]

If putting the outbound thread in the same process as the inbound process is undesirable for whatever reason you can still improve the throughput with what I refer to as a TCP/IP hop.

In the attached screen shot the outbound hop/send thread (hs_pahtnet_12301) is in the inbound process (ib_pahtnet_8015) and sends via TCP/IP to localhost (127.0.0.1) listener hop/receive thread (hr_12301_pathnet) which is the inbound thread to a completely different process (hr_12301_pathnet).

This architecture increases overall throughput dramatically because now both processes are able to act independently of each other.

If you are smart and limit the stuff in the inbound process and divide the work amoungst multiple outbound process groupings you will be amazed at the throughput increase.

Personally I draw the limit at 8 or nine threads in an outbound process grouping at most but less is even better, while I try to keep the inbound process groupings streamilined to one inbound thread and mostly hop/send threads or just a handfull of high priority outbound threads.

This technique is very effect when the source system owners are calling you about messages queing on their system because they are generating them faster than Cloverleaf is ACKing them.

This was the case with this site before I utilized this technique, messages backed up badly on Pathnet during peak times.

Now there is never a backup on Pahtnet and I’ve seen as many as 40 messages a second come across which is a blazing speed I could never come close to with the architecture we had when I first came to work here.

I must admit I learned of this approach by reading Clovertech and it has been one of my best finds for reliveing message flow starvation and bottle necks during my journey.

Russ Ross
RussRoss318@gmail.com

[Ivan Ng]

Russ, Thank you for going along with this thread.  Have tried the TCP/IP before we tried translation throttling.  I believe that with both methods used, the performance would be much faster.  But we have decided against it becuase it will make tracing the message miuch more difficult.

If a message failed at the outbound, we will use DB Administrator to get the source MID of the message and use SMAT to get the raw message back.  With this setup we won’t be able to trace back to the source message.

[Author]

Replies