auto-bounce thread based off alerts not very clean

[Luke Anderson]

Because of network time-outs/firewalls that drop connections in such a way that Cloverleaf doesn’t know the connection has been cut, I have been trying different ways to auto-bounce a thread based off a lastr alert in our Cloverleaf 5.8 but so far, have not found a very clean way of doing it.  In researching this, I have found the following:

1)  The standard hcicmds for pstop and pstart don’t work from the drop-down List when configuring an exec alert line using the GUI alert tool

hcicmd -p p_other -c “problem_thread pstop”

hcicmd -p p_other -c “problem_thread pstart”

But I saw via another user’s Clovertech task that these don’t work anyway.  They just look nice I guess.

2)  I setup a bat file to execute the pstop and pstart based of the lastr alert and this works about 50% of the time.  The other 50% of the time it leaves the thread in a “down” status.

3)  To fix this secondary problem I have setup two alerts, the first one is a lastr that calls my auto-bounce bat file that leaves the thread in a “down” status about half the time.  Then I setup another alert – a down alert – that calls a pstart bat file to bring the thread back up.

All in all, not very clean.  Anyone have anything better?

Topic

[James Cobane]

Luke,

You should be able to do this via an alert.  I would recommend putting the stop & start as one exec action (separated by a semi-colon).  That way the start will execute after the stop has completed.  Simply choose the exec action, and enter the standard hci commands in the Command dialogue box –

hcicmd -p p_other -c “problem_thread pstop”;hcicmd -p p_other -c “problem_thread pstart”

Hope this helps.

Jim Cobane

Henry Ford Health

[Luke Anderson]

Thanks for taking a look James but I just now tried that and nothing happened.  The alert fired because I received an e-mail but the thread status never changed (this connection always changes to “opening” after you bounce it) and there is no evidence that the thread was touched in the process log.  I guess I’ll keep trying some other things.  Thanks.

[James Cobane]

I would review the alerts.log ($HCISITEDIR/exec/hcimonitord/alerts.log) and see if there is any indication in there if the exec action failed.

Jim Cobane

Henry Ford Health

[Luke Anderson]

I checked the log and it did not show any activity.  I am having some luck getting the lastr alert to exec an hcicmd if I enter it as follows with “nt” added onto the command:

hcicmdnt -p p_other -c “cellnetix_r pstop”

hcicmdnt -p p_other -c “cellnetix_r pstart”

(I found this “nt” trick in another Clovertech entry – seems to be the only way to use hcicmd’s from tcl within tcl procs and also these alerts).

I also have the standard e-mail action sandwiched in between these two actions.  This worked once but the alert still leaves the thread in a “down” status sometimes.  The log shows “Thread is already running ‘problem_thread'” when this happens.  So it’s like the engine doesn’t know the status of the thread by the time it execute the pstart command and thinks it is still up.  As I noted before, this isn’t very clean but I’m going to do some more testing.

[Henry Bauer]

I have been able to successfully execute stop and start commands from an alert I created. I have set it up in this fashion.

exec {hcicmd -p hbo_test -c ‘to_nscall_adt pstop’}

tcl{sleep 10}

exec {hcicmd -p hbo_test -c ‘to nscall_adt pstart’}

It seems to work very well with no problems(so far).

[Russ Ross]

Almost every alert we fire for an interface calls our KSH script to cycle the thread and I never noticed the issue you described.

I see we have a 5 second sleep between the stop and start which might help with your problem.

We are running cloverleaf 5.6rev2 on AIX 5.3.

Here is the code snippet of interest

#————————————–
# define functions local to this script
#————————————–

function recycle_thread {

[code]#

---

# define functions local to this script
#

---

function recycle_thread {

Russ Ross
RussRoss318@gmail.com

[Luke Anderson]

I was wondering if you thought that the sleep command you are using would put everything running in the command thread to sleep as well for that brief period of time?  I thought I saw something about this in another Clovertech thread on a different subject but I’m not sure if that could be potential problem or not.  I am now able to control the thread consistently by using an alternative to the sleep command as below in my .bat file:

hcicmd -p %1 -c “%2 pstop”

REM Wait for 5 seconds before re-starting thread

ping -n 5 localhost > NUL

hcicmd -p %1 -c “%2 pstart”

The ping command is keeping the thread down long enough so that when the command interpreter comes to the pstart command, it can now see that the thread is actually down and executes the re-start consistantly.

Thanks all.

[Paul Bishop]

since you’re on 5.8, you could also use

hcicmd -p $process_name -c”$thread_name prestart”

instead of the pstop, sleep, pstart.  then cloverleaf will take care of the timing for you.

Paul Bishop
Carle Foundation Hospital
Urbana, IL

[Russ Ross]

I don’t think the sleep invoked from the alerts puts the process to sleep that I’m aware of.

If something called from within a cloverleaf process like a TPS proc does a sleep that will put the entire cloverleaf process to sleep.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

your post on using a ping to do a sleep sounded familar to me.

I looked and my perpetual looping bat file to ping once a minute to keep my VPN connection from timing out, which uses the ping to sleep 60 seconds before pinging again.

You would think the DOS operating system would have a sleep but I ended up using the ping, too.

@echo off
:START
[code]@echo off
:START

Russ Ross
RussRoss318@gmail.com

[Author]

Replies