problems with 5.4.1 rev2

[Kevin Scantlan]

We are currently on Cloverleaf 5.4.1 rev1 with AIX 5.2 and we installed rev2 yesterday morning.  We had been running rev2 in our test system for a couple of months.  It went fine until we started getting our regular load of ADTs from our IDX system.  They started backing up something fierce.  Our disk I/O was 98%-99% all the time and we were processing inbound ADTs twice as slow as usual. The day before our disk I/O never got higher than 55%.  Needless to say this was having a major impact on the institution.  We called support and they looked around and didn’t see anything and so suggested that we back out the install which we did.  When we came back up we still were not processing any faster and the disk I/O is still high.  So I take down some threads that are in the same process as our inbound thread and the processing jumped up.  Now, mind you, we have plenty of CPU and it wasn’t being hit hard.  But then the thread I had taken down started queueing up in front of it and so we had to turn it back on.  But I would take a different thread down and the processing stayed up.  In either case, I worked it this way all morning and afternoon until the some 15,000 messages queued up on the IDX had finally caught up.  

The real test will be this morning to see if the disk I/O stays down and we’re able to process the IDX messages at our normal speed.  So far it looks good.  By this time yesterday we were drowning.

So, it appears that the problem was with the rev2 patch.  Has anyone else experienced anything like this?  Support says no one has told them of any problems.  

Speaking of support, we were very disappointed in their response.  At the first call when they recommended backing it out,  it was ok.  But we called back after we backed out and were still having problems, and their response was that they didn’t know what to do.  They simply asked what do you want us to look at?  After giving them the symptoms, I would have thought that they would want to take a look, would have some ideas, and have some suggestions for diagnosing the problem.  And if they didn’t have the expertise, then bump the problem up to the next level.

Anyway, anyone with any similiar problems?  What do you all suggest we need to look at?

Topic

[Bob Richardson]

Greetings,

In looking at our production system today rather than watching football, (yeah right), I came across your problem.  We are running Cloverleaf 5.3 on Aix Unix 5.2 with the 5.4.1 Rev2 engine doing our X12 stuff only. We have done some benchmark performance testing with 5.3 Rev3, 5.4.1 Rev2,and  5.5 Rev1 (with our HL7 stuff) and found serious performance degradation up to 18% over the later versions to 5.3 !!  In looking at the Release notes for these later versions, bulkcopy has been repaired (fixed) for issues like copying from a 2.2 variant to a 2.3 variant without blowing up; also, some memory leak issues).  It is our working hypothesis that the Bulkcopy is slowing down our engine:  the Tcl interpreter 8.4 is faster but the Bulkcopy (and presumbly Pathcopy) use C code modules and not TCL to do their work.  There is an official case number on this performance problem.  Maybe you use Bulkcopy and Pathcopy extensibly as much as we do and that is dragging down your engines now with the volume?  Just a thought for you to consider for cause here.

Please post if you do come across root cause for your performance problems as we are interested in establishing our migration path for Cloverleaf integrator.

[Kevin Scantlan]

We didn’t have any performance issues until we went to rev2.

[Author]

Replies