Time out when issuing hciprocstatus or hciconnstatus

[Jon Blanchard]

We have one server left that is running version 5.4.1 on AIX 5.  Any call to hciprocstatus or hciconnstatus results in a timeout trying to contact hcimonitord.  The inability to make a successful call is preventing the archiving job from completing properly.

hcienginestop and hcienginerun both work.  I checked the output from ps -ef for all instances of hcienginewatch and there are no dupicates for any processes.  

I have not gone as far as to purge the pid and wpid files, in case a port conflict exists, but was not sure if that would help.

Your thoughts are appreciated.

Migration Consultant
Coffee Regional Medical Center
Eastern Time Zone

Topic

[James Cobane]

Just for giggles, you might want to cycle the monitor daemon.

Jim Cobane

Henry Ford Health

[Jon Blanchard]

Already done, with no improvement.  Forgot to mention that…sorry.

Migration Consultant
Coffee Regional Medical Center
Eastern Time Zone

[Chris Williams]

Jon,

You mentioned that you are concerned with purging the pid and wpid files,  The files pid, wpid and cmd_port are deleted each time a process stops and are created each time a process starts. A process will refuse to start if any of them already exist, and the error message will indicate that the process cannot be started because it is already running.

Cheers.

[Jon Blanchard]

Thank you…your answer coincides with the lack of any startup issues in any process on the engine.  That puts me back on square one…what am I missing?  This server has enough memory to make hardcore gamers drool (32GB) and handles the interface communications for five sites used for three separate hospitals.  Neither command works for any of those sites.

Do hciconnstatus and hciprocstatus connect to hcimonitord through a specific port and, if so, which?

Migration Consultant
Coffee Regional Medical Center
Eastern Time Zone

[Jim Rawls]

Jon,

Is the process hung?  We saw this sometimes in the early 5.x releases when we were running on AIX.

[Jon Blanchard]

For the particular engine site that I was working on, none of the processes were hung; I cycled all the processes in that site as well as monitord and lockmgr and did not notice any issues with bringing processes down or up.

I will try the same thing with all the engine sites and processes on that server, to see if any process is causing problems.

Migration Consultant
Coffee Regional Medical Center
Eastern Time Zone

[Levy Lazarre]

Jon,

I just had this issue today (hciprocstatus command timing out). What worked for me (and has worked every time) is to reset the host server:

Code:

hciss -k h

hciss -s h

In fact, I couldn’t even open a shell window from the GUI. I had to telnet to the Cloverleaf server from a DOS window in order to issue the above commands. Once I did that, everything started working properly. Not sure why but I have to restart the host server every few weeks for the GUI to remain quick-responsive.

I hope this helps.

[Jon Blanchard]

Latest news:  I cycled all processes, hcimonitord and lockmgr for all sites on this engine, but both hciprocstatus and hciconnstatus still indicate a timeout while attempting to connect to hcimonitord, even after using the setsite command.  Another odd behavior that I noticed is that the AIX netstat command is not working either; it runs very slowly.

There must be a hung port somewhere.  Both hciprocstatus and hciconnstatus invoke the hcitcl function makeStatusServerConn.  Does anyone know what port or connection method this function uses?  Is Monitord the status server or is there another background process that should be running?

Migration Consultant
Coffee Regional Medical Center
Eastern Time Zone

[Jon Blanchard]

Epilogue:  

When hciprocstatus or hciconnstatus is called, the engine software checks its environment by obtaining the IP address associated with the servername.  In this case, the servername was tripwm12; when the engine software attempted to identify or obtain the IP address through a DNS lookup, our corporate DNS server had the wrong server name (trista12) associated with the engine IP address.  That is how the McKesson support person explained it to me.

So hciprocstatus and hciconnstatus were trying to ping the wrong IP address, or no IP address at all.  Once the servername was corrected on the DNS server, the two proc calls worked correctly and quickly.

This issue is now resolved.  Thanks to everyone for their contributions.

Migration Consultant
Coffee Regional Medical Center
Eastern Time Zone

[Author]

Replies