@todd-lundstedt
Forum Replies Created
-
AuthorReplies
-
You’re welcome, Gena. That actually happens far more often, here, then anyone would like… sometimes, you just want to bean a vendor in the head, ask them “what, exactly, were you thinking?”, and then hand them their sign (referencing Bill Engvall’s comedy routine). 😆 I am glad you got it all figured out.
Check your Cloverleaf servers’s netstat output for connections on that port after it “goes back up”
Windows:
netstat -an | findstr
Unix/AIX:
netstat -an | grep
Verify the connection is coming from the IP address of the sending system. If it is, and if the sending systems interface program is still down, they have something else running and acting as an interface on their server. If the IP address is coming from some other system, check that system to see why it is trying to communicate with Cloverleaf. Perhaps their “old” version is still running interface programs.
If you didn’t mention that you see the connection go down, and then go back up (instead of just staying up) when they stop their outbound to Cloverleaf, I would suspect their interface shutdown methods don’t disconnect correctly, and the Cloverleaf “server” thread continues to think the connection is up and good. But, because you mentioned you see the connection go down, I suspect some other program on that sending system, or a program on some other system is configured to use your threads IP/Port for interface communication (just with no data to send, hence the supposition it’s their old version).
On AIX 5.3 running Cloverleaf 5.5..
hcihostid returns…
b2314d6
uname -m returns…
000B2314D600
lscfg -vl ent0 returns…
hciconnstatus will get you the up, opening stuff.
Tom Boyd wrote a TCL script to get status, queued, last r/w times, etc… and even a monitor mode where you could monitor one specific thread. I honestly don’t recall how I came to possess it, because I can’t find where he posted it on Clovertech (in a quick search). He called it “tomst”. You might ask him if he can send it to you.
I need to do some work on it, because it bombs off if a thread name begins with a number.
You can also test your ACK code, and engine code speed this way…
On a test site, set up four threads (adjust ports as necessary):
in_thread_a, server, port 8000, process1
raw_route to
out_thread_b, client, localhost, port 8001, process1
in_thread_b, server, port 8001, process2
raw_route to
out_thread_a, client, localhost, port 8000, process2
Make sure you have your ACK code in place. Resend a dozen or so messages in one of the outbound threads and watch them go round and round. Take notes of your msgs/sec, processor utilization, and any other metrics you want. I don’t think you could ever achieve these numbers in real interfaces, as you have taken out the majority of the network, and any other application delays, but it will tell you just how fast your ACK code can ack while running in an engine.
Stop it all and turn on save-messaging on all of those threads, and let it run again (but not for too long, or you’ll fill up your disk!). Note the differences in msgs/sec when Cloverleaf has to save stuff to disk. With the .idx files, you can look at the time deltas between batches of message sends and report just how fast the ACK code works.
I used this setup when I was testing cluster failovers prior to implementation of a new cluster to help verify we would not lose a message during a failover. I tried it with and without ACK code in place. Without the ACK code, I only ever lost one message per failover.
On the other hand… as mentioned by others here, a sniff of the packets and appropriate analysis will tell you lots, too.
Garry,
Yes, there is a little wrapper script that runs when the cluster resource group starts, but I don’t know what it does… I wasn’t involved until after the cluster was mostly configured (incorrectly, but I had them correct it to a more sane configuration). I might try the Cloverleaf consultant who did that install, and see if he has a quick answer.
Max/Robert,
Yeah, yeah, yeah, I know… but dagnabit… if they want me to work in Windows, I better start learning ways to deal with it. It’s not nice to bring someone along to translate my speech when I am invited to dinner… my translator has to eat, too. 😀 🙄
I’m not going to be able to install these converters everywhere, so I will just stick with a .bat job that sets the environment before calling the powershell script until I figure out how to set the environment variables from withing PS, or I am shown how.
Mark,
Yes, I know I can do that. I assume it is OK to reference a non-existent drive in a variable statement (the drive may be on the other server). However, that suggestion will be a last resort, as I want to code this to work across Cloverleaf upgrades without changes to the OS. Therefore, I would run a batch script to perform a setroot and then execute the Powershell script prior to hard-coding environment variables at the OS level. I was just hoping there was some way get a “setroot” execution from within Powershell to obtain the variables; it’s a learning exercise more than anything.
So far, it looks like the .bat script will be my Obi-wan “only hope”.
Russ,
NICE!!! I hope you and your co-worker don’t mind if I steal that. 😛
Thanks, Garry…
This implementation runs on a MS Windows Cluster; at server startup, the disk resource containing the Cloverleaf folder is not available, so I would not be able to “setroot” then. This “batch script to set environment variables at the server level” (so any command prompt from that point out would have those variables) would have to be run during the resource-group start.
Not being a Windows expert, how would I run “setroot” during the resource group start up to ensure all the variables get set for future command prompt sessions? To my understanding, when you open a command prompt and run setroot, all of the environmentals are lost when you exit the command prompt, and are only persistent for that command prompt session.
Wow! That’s crazy! 😯
What have you done as a work-around… a straight telnet and start the hostserver from that session?
Please update this thread with their fix. I am on 5.5, starting to plan an upgrade, and want to ensure we would have a fix, or work-around prior to implementing 5.8.
Thx!
We had similar issues here with XML into SoarF. No kidding! I feel sorry for the HCI, Cloverleaf, healthcare.com, Quovadx, Healthvision folks. They go through more name changes and parnerships than Lana Turner!
(I was gonna say Liz Taylor, but, she is only number 3).
<a href="http://www.seniorsforliving.com/content/article/top-10-most-divorced-celebrities-/131/” class=”bbcode_url”>http://www.seniorsforliving.com/content/article/top-10-most-divorced-celebrities-/131/AIX 5.3, Cloverleaf 5.5 We only reboot the OS for hardware/software maintenance as required by the action being performed (many of our hardware maintenance items are hot-swap).
November 16, 2009 at 11:11 pm in reply to: Threads/Process show Dead in the ide, but they aren’t really #69795We are on 5.5, and this happens quite frequently, typically on only the smaller sites, and typically only on one site per server.
We just issue the refresh command on the GUI (Ctrl-R or File-Refresh (I think)).
That gets all of the threads reporting properly for us.
I would have to confirm this for 5.5, but I think it occurs for any container object, so IFs, ITERATEs, etc. We are on 5.5, and we have the bug where if you copy and paste an IF statement to a location, absolutely none of modifications to the actions within the IF are recorded. The original pasted actions are all there, and I think changes to the IF logic itself get saved (but I would have to try that again to be sure). But making changes to any of the actions within the IF are lost. You have to save the file, and open it up again. I always save the translate, close the translate tab, and open a new one. If I have lots of IF logic to add/change, I copy/paste them all throughout the xlate, save, close, open and start making my changes to the nested actions.
Love the one your with, eh? 😕
-
AuthorReplies
