Engine server spiking at 100%

[Mark Brown]

We are experiencing an issue that cropped up about a month ago and has progressively gotten worse.  We’re running CL6.0.2 on a Windows 2008 server (we’re upgrading soon). What I’m observing is that when an ADT message is sent from our Cerner Millennium system to the interface engine, the CPU spikes to 100%.  We have several other feeds from Cerner and  none of them behave this way.  If we shut down this feed, the engine stops spiking.  This sustained spiking is really bogging down the  engine. Infor is puzzled. Cerner is puzzled and management is unhappy.

Our server guy say it’s not the anti-virus/anti-malware software and he has excluded the CL folder from scanning. I’m lobbying to have it shutdown for trouble-shooting.

Any idea what could cause this spiking to happen?  No ADT from Cerner and it all runs smoothly.

Thanks

Topic

[Keith McLeod]

Could you possibly have a loop in your code?  Or something where you control your loop with discrete indexes as opposed to relative indexes?    I once had code where I needed to pull a certain index from a list and found that whenever an & was used it created another split and moved the values associated with the index.  Ultimately processed but was fundamentally wrong when all was said and done….  Just thoughts.  Could be in translate or tcl….

[Jim Kosloskey]

Echoing Keith’s thoughts – I would investigate all Tcl code (UPoC or Xlate extensions) including inherited code or even code provided with Cloverleaf.

Perhaps turning up the engine Output to ALL will give a clue as to what module is executed prior to the spike.

Does this also happen in test or is it just production?

If not happening in production, I would compare all configuration and code to see what is different.

More investigative work needed I think.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Mark Brown]

This is happening in production. About the time this started happening we had gone live with a system called Ignite. it creates a huge alias in PID:3 that exceeds the HL7 spec for length . They provided a script to truncate this field.  We hadn’t installed it. But now we did and it has no impact.  Infor support has no idea what could cause it

[Jim Kosloskey]

So does this integration also exist in a Test environment?

If so does the spike happen there as well?

If not – what are the differences between the Test and Production environment?

I really doubt a too long field in a message would cause this truncated or not (unless the message length itself is excessive).

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jay Clements]

Check the sizes of your databases in /exec/databases for each site. Anything over 500mb is unwieldy – according to Infor. We had an issue where one got to nine gigabytes do to an upstream change we had no knowledge of (accumulating data at the rate of 1 gb a day). First thing we noticed off was CPU spiking. Hope your issue is unrelated because it was a pain to figure out and fix.

 

 

[Jim Kosloskey]

Rob – good input very on topic.

Mark this is probably a good thing to check.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Author]

Replies