Xlate vs Tcl Filtering – anyone quantify performance difference?

[Jim Kosloskey]

Has anyone out there done the exercise of quantifying the difference in performance of Filtering messages using Tcl versus Xlate?

If so, are you willing to share your methodology and results?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

Topic

[Vince Angulo]

This was long time ago (v5.4), and I don’t have the detailed results anymore, but here’s my recollection.

What I did was create a few file to file interfaces.  One with a tps proc to filter on ADT types, one using an xlate, and one using a reg exp in the routing.  I believe I was filtering out merge and delete transactions (A34/35 and A29).

Created a file of 20K messages, ran them through each interface. This was on our test server with almost no other interfaces competing for resources. (AIX).

If I recall correctly, the tps proc was faster by several percentage points.  Routing edged out xlate, but only modestly so.

[Jim Kosloskey]

Vince,

Thanks for the response.

Do you recall what metric in the engine you used to measure the performance or was it an observation of something like elapsed time?

 

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

I set up a test for this situation and I am focusing on IB Latency as a valid metric. However, I am getting numbers that only make sense if that figure is calculated not with each message but rather over a scope of time.

The documentation indicates IB Latency is the time (in milliseconds) from the time a message is a read and when it is translated.

In my test I have a file with 40,000 small (10 character) messages with a message that will pass filter every 1000 messages with one at the end that will also pass. Thus, 41 messages out of 40,000 that will not get filtered out.

So, in the case of Tcl filtering, only 41 messages reach the Xlate, while with Xlate Filtering all 40,000 reach Xlate but only 41 pass through.

In most of the tests (I have run multiple of both Tcl and Xlate filtering), the IB Latency for the Tcl Filtering is higher than the IB Latency for Xlate filtering.

To me this means IB Latency is calculated not by summing the time for each message that gets to translation then dividing by number of messages but rather taking a scope of time (say from thread Up to Last Read) and dividing by the messages that made it to Xlate.

If the scope method is used, then when Tcl filtering (in this test case) is used only 41 messages make it to Xlate AND if the time scope is divided by 41 that will produce a larger number than the relatively same time scope divided by 40,000 messages. That would result in the Tcl Filter IB Latency being higher than the Xlate Filter IB Latency which is what I am generally observing.

Does anyone know for sure exactly how the thread Status IB Latency is calculated?

Where is my thinking above faulty?

Thus far, using IB Latency and the above observation, I see approximately a 2% difference (Tcl filtering 2% faster) or (on my system) 2.2 milliseconds.

When I have completed the measurements, I will produce a report with my methodology and the test results. I will also have the Xlate that creates the test file; the Tcl proc; the filter Xlate(s); and the NetConfig setup for both processes available for any who would like the results.

If someone would like to work with me on this as I move forward, I welcome that involvement. Just email me.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

OK – I found out why my numbers were skewed (Xlate many times being less than Tcl). I had different throttling numbers for the tests. For Tcl I had 1500 messages read, reading every 1 second; for the Xlate I had 1000 messages read , reading every 2 seconds.

Once I mad the Xlate the same as Tcl and reran the tests, more often than not the Xlate IB Latency was higher than the Tcl IB Latency.

The numbers in each category bounced around a bit which I think is an indication my system was not as quiesced as I would like but as quiesced as I can make it. After eliminating the outliers, here is a summary of what I observed for the test as I set it up (based on using IB Latency as the comparison metric):

Xlate Filtering vs Tcl filtering –   14.27% slower on the average.

Xlate Chain Filtering vs Xlate Filtering – 3.5% slower on the average.

Xlate Chain Filtering vs Tcl Filtering – 18.42% slower on the average.

I think I need to have a larger sample as the number of outliers equaled the number of ‘reasonable’ results leaving a sample size of only 3 tests. More samples I think are necessary as well due to the obvious hidden activity in my system which resulted in a number of outliers that need to be scrubbed from the population.

But the above is a good starting point.

I also need to see if I can make my system more quiesced so less outliers are experienced, and I can have greater confidence in the numbers observed.

I still would like to have a clear understanding of how the IB Latency metric as shown on the NetMonitor Thread Status is calculated so if anyone has some insight on that, I would appreciate it.

Again, the goal here is to attain a performance relationship between Tcl Message fltering, Xlate Message Filtering, and Chained Xlate Message filtering. That knowledge can then be used, along with other considerations, to make a decision as to what method of Message filtering to use.

There are Cloverleaf shops which do not have the Tcl expertise to do filtering via Tcl and Xlate filtering becomes attractive. But that decision is possibly made in a vacuum (lack of knowledge of the relative cost associated with Xlate Filtering execution).

The intent is not to advocate one method over another but to try to put some metrics on the options.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

Here is what I think I have observed about the IB Latency metric.

The Documentation is not wrong but I think not nearly expansive enough.

The latency measurement is from Message read until ‘AND IF’ the message makes out of the Translation Pthread. Thus, messages which are filtered at the Inbound Tps, Pre or Post Xlate are NOT included in the metric. It also appears the measurement is indeed made per message then accumulated for presentation in the Thread Statistics. So, the presented metric is the IB Latency for all messages that are read and make it to the Outbound thread.

Obviously, this metric is not useful for what I need – that is to measure the relationship between Tcl filtered messages (typically filtered either at the IB Tps UPoC or Pre-Xlate UPoC) and Xlate filtered messages.

So, I switched to the elapsed time expressed as the Up time and Last read. This means some gyrations in the testing. It also means the metric used is not precise enough to give more than a basically accurate representation of the time for each message, but it is sufficient for evaluating the relationship among Tcl Filtering, Xlate (with no chain) filtering, and Chained Xlate Filtering.

I have collected the results from 42 tests and now will work on the evaluation report.

I will report back when that is complete and anyone who wants a copy can have one.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jim Kosloskey]

I have finished what I think I can accomplish, and the report is available.

The general summary is: Xlate Filtering is slower than Tcl Filtering (no surprise) and Xlate Filtering with chain is slower than a single Xlate Filter (again no surprise). The differences on an average and percentage and seconds per message though was interesting (but not overly surprising to me).

The details, observations, and summary are in the report.

If you would like a copy of the report, email me.

You can use the report as you see fit as long as you leave my name in the trailer and give attribution when referencing the report.

What I would like from you when you get the report is to do the same testing on your system and provide the results and report to me. I will incorporate those results in a larger report with attribution to you.

I will provide the materials you need to do the testing and reporting (with the exception of you have to provide the Cloverleaf environment and platform it runs on).

I would really be interested if you found a more representative metric than I used.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies