Unkillable Cloverleaf Zombie Process

This topic has 0 replies, 1 voice, and was last updated 6 years, 4 months ago by Jay Clements.

Creator

Topic
August 7, 2018 at 2:31 pm #55761
Jay Clements
Participant
Cloverleaf: 6.1

OS: Windows 2008 R2

Over the last week we have had two instances of site processes locking up and becoming unkillable. When I say unkillable I mean that the only way to remove the process is to reboot the integrations engine. Since we already do a monthly reboot during our regular patching having to do two reboots, so far, outside of that schedule is not sitting well with me or management.
We have contacted Infor and their tech walked us through rebooting, but at that time didnt really have any idea of why this would be happening and what a longterm solution would be, or at least we did not discuss it.

When a lockup occurs it becomes impossible to bring an effected process back up. We have to use “hcilmclear” on the process to break the lock manager and then we can bring the process back up. it is at this point that the process duplicates. The new process will also stick if brought down, however. We cannot complete our standard shutdown and cleanup procedures. see image below

This process can not be killed, either through task manager, command line, powershell, anything.

further investigation seems to lead to a hung thread within the process itself, but even trying to suspend or kill that thread fails. we have, out of curiosity to know more about this behavior, watched it overnight. The new process hums along just fine but the zombie process remains static and unkillable. The only solution appears to be a reboot.

We’re not really sure if that apparent hung thread is related. Xalan is an apache product that implements xerces-C, which is apparently used in Cloverleaf for XSL transformation. Could just be nonesense and completely unrelated. Clearly we are grasping at straws trying to track down any useful information.

Log text follows at the end of this post. This follows the original initialization of the zombie thread – what I mean by that is that PID 5128 was created on 8/4 and has been stuck since then. Succesive attempts to init a new PID for that process results in the logs below.

Any thoughts?

Edit: I noticed TST-SEC files being generated and I am not familiar with their function. They were generated at the time of the last issue and are all locked up. please see image.

Thanks

dbi :dbi :ERR /0: