hostserver problems

[John Mercogliano]

We are having problems with the hostserver hanging after running for a while.  Sometimes it last for weeks, other times we have to bounce multiple times.  I’ve noticed that overtime the thread count for the hostserver process keeps increasing until it hits around 125 which is around when it stops accepting connections from the clients.  

I was wondering if anyone has encountered this problem before or might know why the hostserver would not be releasing threads.  We have about sixty sites with 5 developers using the gui’s to access the sites.  

We are running 5.2.1P2

Thanks,

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

Topic

[LeeAnne Kardas]

I had problems with the hostserver when we keep Cloverleaf up and running for a long time, i.e. 4 months or so.  I incorporated into my monthly maintenance process to stop/start the hostserver once a month and since we started doing this we had not had a problem.

[Russ Ross]

In the past I had hostserver issues which just about drove me crazy.

They might not be exactly the same as yours but now I can run the hostsever for as long as I want and no longer have issues.

I wrote the following KSH script (start_hs.ksh) that I run from an xterm to start the hostserver.

Code:

#!/usr/bin/ksh

cd $HCIROOT/server/logs
myDISPLAY=$DISPLAY
unset DISPLAY
hciss -s h
export DISPLAY=$myDISPLAY

My big key to success was going to a directory the hostserver could write to and unsetting the DISPLAY before starting the host server.

We are currently running Cloverleaf 5.2.1P2 under AIX 5.2.

If interested here is the corresponding KSH script (kill_hs.ksh) that I run from and xterm to stop the hostserver:

Code:

#!/usr/bin/ksh

cd $HCIROOT/server/logs
hciss -k h

Russ Ross
RussRoss318@gmail.com

[Kevin Scantlan]

Ross,

What kind of symptoms had you been seeing?

[Kevin Scantlan]

oops… I mean Russ…  🙁

[Russ Ross]

The symptoms I experienced were that the hostserver would consume more and more memory until I was forced to cycle it.

Then when I would try and cycle the hostserver, I could kill it but had to try numerous times to start it before it would stay up.

To monitor the size of memory usage I invoked topas -P then cursored to the DATA RES column to monitor the memory usage and that darn hostserver would never stop growing and even got bigger than 1 GB once before I recycled it.

I just did a check of the hostserver memory consumption on both TEST and PROD and they are both consuming less than 8 MB.

I no longer give any thought to needing to recycle the hostserver as a necessity.

Now I only cycle the hostserver to pick up a chnage or during a fail-over.

Here is a screen shot of the topas with the hostserver entry higlighted.

Russ Ross
RussRoss318@gmail.com

[John Mercogliano]

I have figured out the problem and it is now CASE#31278 with support.  

What I found was that we left a couple process out of the cycle log script we used.  When the log file got to big and you viewed the site with Netmonitor the hostserver would open the file and freeze the hostserver.  I was able to repeat this a number of times.  Once I cycled the log file the hostserver stopped freezing when I looked at the site in netmonitor.  According to support the only file that should be opened by the hostserver is server.log

This is how I determined which site was the problem:

I currently run HP-UX 11.11 and QDX 5.2P2

1. run gpm(glanceplus)

2. From the reports menu select process list which opens a new window.

3. Select the hostserver process. In this program it is listed as   PA_RISC2.0

4. From the process list reports menu select the process open files which opens a new window.

5. This lists all open files associated with the process.  I saw some file offsets with very large numbers associated with one of our log files.  After cycling the connections log file and restarting the hostserver the problem stopped.  

Russ, I will try your script also, as we also have problems with needing to cycle the hostserver because of memory size.

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Russ Ross]

Even though we have an in-house standard requiring all our prodcution NetConfigs to cycle process logs that become too large, we also overlook them occasionaly.

Fortunately we monitor our filesystems via “df-KI” and noticed the problem before it became critical.

Here is a script (check_disk.ksh) I wrote to help me locate the big offenders quickly.

Code:

#!/usr/bin/ksh

echo “Megabytes   Directory”
echo “———   ———”
du -k | sort -n | awk ‘{printf(”%9.2f   %sn”, $1/1024, $2)}’

You can cd to a filesystem of interest or directory and then run it at command prompt to get a listing of the entire diretory tree sorted by disk consumption size.

I have found this very usefull to quickly narrow my search to where I need to focus my attention.

Remember this works on AIX and might need a slight modification to run on another flavor of UNIX like HP.

Russ Ross
RussRoss318@gmail.com

[Author]

Replies