Looking for advice from high volume CloverTechers…

Hi all…my company is using Cloverleaf to replace the current data acquisition layer which is a bunch of Python code that has to be customized for each new customer.  While Cloverleaf has numerous advantages (scalability, implementation time, maintenance, logging, troubleshooting, etc.), I’m worried we will be a performance bottleneck in our goal of processing millions of transactions per customer per day.

Our setup is fairly simple (though I’m open to suggestions here too).  One process with one thread that reads in .csv files from sFTP, processes through an xlate depending on the file name and creates .json objects, then sends to another thread in its own process that sends the data outbound.  I see  the file get read in fairly quickly but then the “pxqd” builds up and slowly gets dequeued.  If i take the average of “Xlated”  by “Xlt Time” things seem pretty fast but that does not seem to be a true time of how many we are processing per second (or there is a bottleneck between something being xlated and it actually sending outbound)

I’m looking for any advice that other high volume Cloverleaf can provide in two areas:

1) Processing times:  I know everyone has different data with variable amounts of tcl and xlate code, but what settings have you tweaked to maximize performance?  I’ve looked at a lot of the Process specific ones (multi-threading,  disk based queuing, disabling all logging), but I’m not sure which are helping and which actually end up slowing things down.  

2) This may be worthy of a separate thread, but how does everyone benchmark these numbers?  So say I drop a file of 10,000 or 100,000 messages, what is the best way to determine inbound to outbound processing time of the entire file?  Has anyone figured out a way to automate benchmarking and, even better, send the results out of Cloverleaf?

Thanks for reading this long post and any advice you can offer!