Performance issues in Cloverleaf 6.2

[Rob Abbott]

All,

We had a handful of high-volume customers report performance issues with release 6.2.

Rob Abbott
Cloverleaf Emeritus

Topic

[Rob Abbott]

Please note that this material has also been published to the Xtreme knowledge base under article 1967495.

Thank you

Rob Abbott
Cloverleaf Emeritus

[Steve Pringle]

Hi Rob –

Inquiring minds – is there an ETA for the 6.2.2 release?  

thanks,

Steve

[Rob Abbott]

Sure – from the document:

Patch 6.2.2 is expected in late May. This date is subject to change without notice.

Rob Abbott
Cloverleaf Emeritus

[James Cobane]

Rob,

Can you elaborate on what is a “high-volume” customer?

[Rob Abbott]

Hi Jim, it’s hard to answer that because we have seen this impact some customers and not others.

Rob Abbott
Cloverleaf Emeritus

[Bob Schmid]

Gotta understand “complicated”..did TCL suddenly become slower or did the hci extensions…?

hops – ? you mean intersite routing / cross process  or simply localhost to localhost sends client to server tcp/ip?

we have a lot of process to process and site to site client/server localhost raw sends due to the complexity of our enviornments…no way around it right?…unless we being writing to disk

Being as smart as we can……hoping that something is not beyond our control like a BIG fat piece of ENGINE internal

[James Cobane]

We have multiple sites with inter-site threads as well as many ‘destination’ threads between sites.  A fair amount of tclprocs with some using SQLite DBs.  We have over 6.5 million outbound transactions daily, so I am definitely concerned about the performance impact.

Jim Cobane

Henry Ford Health

[James Cobane]

Additionally, I had opened an incident (11262517) on February 7th with Support to notify them of our planned upgrade and specifically asked “…if there are any last minute issues/concerns with 6.2.0.2 that we need to be aware of.”  The response back from Support was “No upgrade issues have been reported yet on CL6.2.0.2”.  When did this issue become known?  It would have been helpful to know of the potential issue sooner.  Also, if a customer has specifically notified Infor of a planned upgrade, it would helpful if Infor would inform the customer of new issues affecting the version they are planning to upgrade to.  I hate to think what the outcome may have been had we moved forward with our upgrade without the knowledge of this issue.

Jim Cobane

Henry Ford Health

[Rob Abbott]

Bob: please read the PDF for the details.  Tcl 8.6 is slower than Tcl 8.5.  Cloverleaf 6.2 uses Tcl 8.6.  We plan to downgrade to Tcl 8.5 in 6.2.2.

Jim: I am doing all I can to present the facts and be transparent.  This technical bulletin was delivered to Support on the same day I posted to Clovertech.  We spent an enormous amount of effort tracking this down.  We went down several dead ends in this process.  I did not want to present misinformation or cause confusion until I had all the facts straight.

Again if you are concerned about this, I encourage you to do benchmarking in your own environment to see how it responds.  I can’t stress enough that we have seen performance impact in a minority of environments but not others.  I have given you facts and what I have observed anecdotally.

If you do not have the resources or time to benchmark, please wait for 6.2.2 for a full resolution.  Your patience is appreciated as we work through this.

Please let me know if I can answer any additional questions.

Thank you.

Rob Abbott
Cloverleaf Emeritus

[James Cobane]

Rob,

Thanks for the information and disclosure.  I appreciate the transparency and have full confidence in your team’s efforts to resolve the issue.  I think at this time I will recommend postponing our upgrade, as I feel we fit the profile of those who have experienced the issue.  But, we will have some internal discussions before making a final decision.  As much as I wanted to get the upgrade completed, I would err on the side of caution.

Attempting to benchmark may give some more insight into if there is performance issue, but no matter how hard you try you just can’t emulate an actual production load or the potential ramifications.

Thanks for your support,

Jim Cobane

Henry Ford Health

[Rob Lindsey]

Thanks for the information Rob.  We were about to start our upgrade process but have decided to wait until the fix is put out.  We are setup exactly as the ones that you mentioned you were seeing issues with (.i.e. lots of TCL, site to site processing, etc).

Rob

[David Barr]

We’re testing 6.2.1.0 and we ran into a weird bug where the engine processes will panic if we start a process that contains threads with queued messages in the recovery database. We’re waiting on a fix from R&D.

May 9, 2018 edit. This is resolved. It wasn’t a bug. Someone copied the recovery database from a 6.1.2.5 site into the 6.2 site. The schema for the recovery DB had changed slightly. It mostly worked except for the error described above. When we reinitialized the database, we no longer had this problem.

[James Cobane]

Any update on the anticipated release date of 6.2.2?  Is it still on target for a late May release?  We are trying to lay out our implementation plans for the remainder of the year.

Thanks,

Jim Cobane

Henry Ford Health System

[James Cobane]

It is now officially the end of May.  Any update on the release of 6.2.2?  I didn’t see anything on the Infor site.

Any update would be appreciated.  We are trying to layout out our implementation schedule for the remainder of the year.

Thanks,

Jim Cobane

Henry Ford Health

[Steve Pringle]

Same here.  It would be helpful to know when the release will be available.

thanks,

Steve Pringle

[Bob Schmid]

Steve…..what release are you running and what OS?

What are your issues

We are beginning our 6.11 to 6.2 planning…….should I hold off?

We are a huge install ~ 25mill messages a day

Bob

[Steve Pringle]

Bob – we’re on 6.1.2, AIX, no issues there.  We run ~12-14 million messages a day, so we’d prefer to wait for the performance fix just to avoid any potential issues.  We also want to move off of Windows 7 as most of our organization has done so.  The 6.2 gui supports Win 7 and 10, which would allow us to test on WIN 10.

–Steve

[Rob Abbott]

Hi All,

Wanted to give a status update on this.  We have found root cause for all performance issues and resolved them for all platforms.  It turns out that AIX was the most negatively affected.

We plan to GA Cloverleaf 6.2.2 on June 8 (please note this is a target date and is subject to change without notice, but the risk is low that it will change).

The 6.2.2 release shows similar or slightly better performance than 6.0 / 6.1 in our internal and partner testing which includes very complex environments.

Note that this release will include a Tcl downgrade from 8.6 to 8.5.19.  Given the issues we uncovered, we have found that the Tcl performance impact is minimal and will look at upgrading back to Tcl 8.6 in a future release.  We have very solid benchmarking processes in place now so will be able to determine how Tcl versions affect performance moving forward.

Please let me know if there are any questions.

Rob Abbott
Cloverleaf Emeritus

[James Cobane]

Rob,

Thanks for the update.  This is very helpful.

Jim Cobane

Henry Ford Health System

[Author]

Replies