Can you have too many pending messages?

[Mark Brown]

I was curious if there are any performance / stability issues from having messages queue up on a down thread?  Is there a “rule of thumb” number of messages that when reached, you probably should save them off to file?

We’re running 5.3.1 on Windows 2003.

Topic

[Daniel Lee]

I guess it depends on how many messages you’re talking about.  We once did a history load through the engine and would not take that route again if we could avoid it.  If you have a thread that consistently gets backed up I would recommend using the “Disk Based Queuing” in the process configuration of the Network Configuration.  Remember, if you do not do these messages are stored in RAM.

From the history load I did I would even steer clear of having too many messages queued up even if you do “Disk Based Queuing”.  Even though the messages are queued to Disk they still have to read the pointers to the messages in RAM.  We had nearly 100K messages queued up and the tread died on us.  When we tried to bring the thread up again our whole engine stopped processing messages for 15 min. while that one thread loaded all the pointers to the messages in it’s recover database to RAM.  Also, once your DB size grows, it will not shrink until you do a dbinit.

[Rick Brown]

Daniel couldn’t have said it any better.  Cloverleaf has the ability to store messages but there becomes a point where it could cause problems.

I always suggest if your going to have a thread down for a long period of time that you write the messages to a file as you suggested.

[Richard Hart]

Mark.

In our experience (AIX 4.3.3 and CL 3.5.5) a queue on a thread of more than 200,000 messages caused issues with the hcidbdump utility when attempting to sort the queued messages.

ie

attempting to grab the data in chronological order.

The thread worked and sent messages, but a stop and start caused a delay – long enough that we gave up and killed the process!

We only allow large queues for applications that are about to go-live and in our setup, an application connection will be to its own Cloverleaf site, so there is little impact on other production sites.

We use SMAT files to save all inbound messages, ‘thread logs’ to indicate how the messages have traversed through the (generally) two Cloverleaf sites and four threads – this assists with ‘lost’ messages.

Before we connect to a thread with large queues, we stop in-coming messages and dump the queued messages, just in case!

[David Caragay]

One other convenient option is to use Cloverleaf’s ability to “forward” msg data to an alternate outbound thread. This alternate outbound forward thread can be set up as a file protocol (or possibly an alternate destination ip address).  We’ve used this in the past and it helps to keep things flowing through the engine when we have an extended downtime on a receiving system.  When everything comes back up, we can take the file created by the “forward” thread and resend it through the original outbound thread.  

The hcidbdump and SMAT utilities can become very problematic when reading/writing/sorting large files.  The “forward” feature is available on more current versions of Cloverleaf.

[Author]

Replies