Cloverleaf having a hard time processing large amt of data

I beleive what you are experiencing is the priority of the inbound thread.  It reads everything in before writing out.  You might be able to add some thread throttling under process configure within you netconfig.  This requires a process bounce to enable.  Or at least this is how I understand it to be.  If you bounce your thread when may messages in recovery database, it may take a long time to show signs of a restart.

Cleanup is much the same on AIX.

This is from Cloverleaf Documentation.  There are some exmaplesin the users guide…

Translation Throttling

Select the checkbox and specify the Minimum,  Maximum, or Percent values. These control how many messages the translation thread processes each time it runs. Under normal circumstances, the translation thread processes all of the messages on its queue. By specifying translation throttling, the number of messages processed is limited. This allows other threads in the engine process to run.

Minimum  specifies that, if the number of messages on the queue is less than this value, then all of the messages are processed. If the queue has more than this many messages, then the percentage is applied.

Maximum specifies the absolute maximum number of messages to process. If the Minimum is exceeded, then the percentage is applied. If the resulting number of messages is greater than this maximum, then only the maximum number of messages is processed.

Percent  specifies the percentage of messages to process if the queue has more than the minimum number of messages.

Clear the checkbox for the translation thread to process all of the messages in the queue.

Throttling is checked in the following manner:

Get percentage of pre-Xlate queue

Check against maximum

Check against minimum

Check that processing is not more than the number in the pre-Xlate queue

For example:

min = 5

max = 10

percent = 50%

If using the Fileset protocol (my preference), you can use the protocol settings to control the number of messages read and how often a read occurs – effectively throttling the inbound. That will allow you to see some progress in Cloverleaf as well as relieve the potential saturation of your process.

We are currently in the process of backloading about 300,000 messages and I find it easiest by handling it with a fileset-local protocol thread. This allows me to set how many messages I want to read and how many seconds to do it in. I’m set at 10 messages every 10 seconds. That’s 86,400 messages a day. Pretty reasonable…

Rehman,

You could probably calculate that based on how long a duration you could tolerate and how fast the process is.

If you are able to process 30 messages per second, then simple math will tell you how long that will take.

All that the throttling will do for you is to allow you to see some progress (also perhaps relieve consumption of memory due to the Recovery DB usage) it really won’t speed things up.

As a matter of fact throttling may lengthen the duration but so what – this is a large batch process.

Once you have done the math you can set some throttling values and then try a test set to see how that works. Adjust as necessary then go.

Just as an aside, personally I don’t think backloading should be done via any Integration Engine.

I have done some backloading in the past.  It can take a very long time depending on the number of messages.  It was not the processing of the messages as much as it was the ACk NAK handshake and network latency to the destination system.  I had approximately 500,000 messages that would have taken 3 days to send to Centricity.  I opted instead to process all messages through the testing tool.  I then wrote some code to create messages named as though they were recevied via the interface  and zipped them into batches of several thousand each. I then had an operator unzip them into the directory for centricity to process allowing for realtime messages to be inserted between each batch.  This allowed for the realtime data to continue to flow in between each zip file extraction causing minimal delay for realitme data.  It took approximately 10 hours under the watchful eye of the operators.

They could not afford to have the production interface essentially down.