Alerts Firing Strangely…..

[Tom Rioux]

We have a situation here that is puzzling and we can’t seem to find an answer for it.   I’ll try my best to explain it.

We have connectivity alerts that go out to application users if Cloverleaf cannot connect to a particular application.  This informs the user to check the interface on their side to see if something is wrong.  We have a particular application, Optilink, where the user is getting alerts.   The kicker is that these alerts are only happening on Mondays, Wednesdays, and Fridays and only between the hours of 3am and 6am.  The alerts are also are across the board at several facilities but they all share the same server.  Seems a pattern is occuring right?  Well, I checked all the logs and every scrap of information on Cloverleaf and the best I can do is confirm that, yep, the alerts are going out.   I see nothing in the logs that confirm we lost connectivity or anything.

We have spoken with the user.  Everytime they check the interface after one of these alerts, it shows that they are connected.

We have checked with our network folks and they don’t see anything unusual happening on those days at those times.  ( I have thought about a timeout situation but that would be occuring all the night if so, and would happen on Tuesdays, Thursdays, and the weekend as well)

We have checked with our server folks and they see nothing happening on the server (like backups or any other type of jobs) that are occurring during those times that would affect connectivity.

Right now, I’m at a loss to explain it to the end user and the best solution I have right now is to silence the alerts during those times.  However, to me that is not a viable option.  Thats a workaround, not a solution.  

Any ideas gang?

Thanks…

Tom Rioux

Topic

[Robert Gordon]

Denial can keep a person awake at night.

When was the last time your ran a hardware diagnostic on the NIC.  There are several factors that can be happening, finding the right solution is not something that is straight forward.  So here is some ideas which you might want to try in order.

1.  Have the network person check the configuration of the NIC compared to the communication rate of the switch or router, yup your network folks might have updated and or upgraded a router and created and off balance condition on the network called collisions.

2.  Check the bandwidth of your NIC and the bandwidth of your backup server, and make sure the settings are synchronized.  If the backup server is a 1000mb connection then your server should be the same.  If not purchase a new NIC and use the new NIC for backups only.

3.  Have the network people check peak bandwidth utilization and optimize and dedicate more resources during high availability.  Some people have switched over to fiber optics and some have not, but then all routers do have a setting which is beyond the default settings from the box which is a yes or no question for the router.

4.  Do a physical segment test, it could be an early indication of network failure.  Consider talking to facility engineering to see if they had any repairs, renovations, or problems in the suspicious areas.  In one place I saw a broken toilet bring down a network two years later, sounds ironic but true.

5.  Check the backup batteries on the router equipment.  Some batteries have to constantly talk to minitoring equipment, if not they cause an electrical surge or spike and sometimes a brownout.  Film the hospital hallways during the incident, make sure to set the camera date and time, set your focal point to include at least 3 to 4 panels (A bulb is not a panel, one panel is several bulbs), hit the record button, tell the nurse or tech person to watch the camera and walk away.  Route the alert to your beeper or phone, wait for the beep and stop the recording on the camera after you recorded the text of the beep.  Go home, rewind the tape, forward to the end, and do a slow motion of the last 10 to twenty seconds of the tape.  If the camera was pointed at your flourescent lights you will catch a slight dimming of the light on the film.  You might have to do this serveral times to prove your point.

6.  Talk to your Radiology department and check the time of last usage for MRI, X-Rays, Cat Scan, and Pet Scans.  All hospitals have special colored plugs for power circuts and consumption requirements.  A simple thing as radio or lamp in the wrong plug can cause a problem especially when powering down an MRI.  You probably heard of cell phones bringing down planes, I have seen a clock radio give a normal person a brain tumor.

[Troy Morton]

You might also check to see if the Anti-virus is scanning or updating the desitnation server during the times the alert fires.  I have seen anti-virus software wreak havoc on applications that are sensitive to other applications reading or locking their files.

Please post what you find, if anything.  I’d love to hear the reason the alerts are firing.

Troy

[Author]

Replies