Karl,
I do not know of any tools specifically designed to do this on Cloverleaf(R).
However, I know there are a lot of tools in the marketplace that claim to be able to do this. I have not investigated any of them lately, but the ones I have investigated in the past are labor intensive and still require much of what follows.
I will share with you a methodology (this is not a solution, just a methodology) I have used successfully in a past life as a Capacity/Performance Planner/Manager.
Identify your peak demand period(s). All other analysis will tend to stem from knowing and utilizing this period.
Notice that may be plural but it could be you just need to know the demand for a primary function from which all other functions progress. For example, in a Health Care environment it is frequently true that the ADT process is the most active and that all other work is somewhat dependent on the ADT process. You need to evaluate how your environment sits.
The actual period can be any time period but for real time I have found looking for a 15 minute period is usually sufficient.
What you are trying to find out is what time period on what day (hopefully just one day in a month) represents the peak demand (expressed as an arrival rate). That is demand being number of messages arriving to Cloverleaf(R) to be handled.
SMAT data can be helpful for determining this period.
Be sure to evaluate lots of time periods and make sure you are including all of the demand that you think is necesary. You need to find a peak demand period that is so representative of the peak demand that it can be used as the period to watch with confidence.
Of course, you need to re-evaluate the peak period from time to time (generally once a quarter seems to be good but once a month seems too frequent and any longer than every six months tends to be too long). The reason you are doing this is because the business demands in your organization which translate into message flow and therefor demand are happening without your knowledge and the period or the intensity of the period can change.
Now you need to set what we used to call ‘happy numbers’.
These are values that are essentially Service Levels that everyone agrees are the point on the service chart that makes everyone ‘happy’. Better numbers and people are still happy worse numbers and people begin to get unhappy.
These ‘happy numbers’ are frequently expressed in terms of message/second, seconds (or portions of seconds)/message, or some other service level that makes sense. Then the numbers typically have two values – the mid point and a ceiling (the ceiling should be set such that reaching it is not a catastrophe but rather an early warning).
Now that the above is all determined, to test a new release, you need to devise a mechanism to get a set of messages from the peak period and resend those at a controlled pace.
You need to control the pace so that you can simulate the expected arrival rate pacing for the peak period. This lets you evaluate if everything is the same on the new release.
But if you design your mechanism for the ability to control the pacing (not have it static) then you can also take this opportunity to see where your ‘breaking point’ is.
Obviously, you will also need a mechanism for measuring the results and expressing those results in your ‘happy numbers’.
Of course the platform you are testing the new release on has to be ‘exactly’ the same as the normal production platform so that the only difference is the new release. This is the part that is not usually achievable.
If you cannot use the same exact platform construct, then you need to devise a method to interpret your results to accomodate for the differences.
For example, if the platform you are testing on is purportedly a better performing piece of hardware then you need to determine the relational difference and discount your results by that amount, If you are testing on a lower performing platform then you need to enhance the results of the test by that amount.
There is more to be done but once down this path you are on your way and the rest probably will come to you as you progress.
This should not be considered a trivial piece of work.
By the way, the knowledge you have gained in your analysis (peak demand period, etc.) and some of the methods and techniques you develop for the upgrade testing can be very valuable in evaluating your day-to-day performance and assisting in predicting capacity exhaustion.
The idea is to watch the peak period closely – any issues outside of that period are likely to be anamolies and although they should be researched, they are probably not representative of a systemic problem.
There is more… much more…
Jim Kosloskey
email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.
.
