Network Scan causes engine panic.

[Debra Downs]

Last Thursday our compliance group had an outside company performing a vulnerability scan on our network.  The scan was at the ‘port scanning’ stage when all three of our engines panic’d and went down:

[cmd :cmd :INFO/0:fr_starCLO_cmd] Receiving a command

PANIC: “str != ((char *) 0)”

PANIC: Calling “pti” for thread fr_starCLO_cmd

---

Scheduler State

---

Thread Events     State      Priority Runnable  PT Msgs

  0      0   SCHED_RUNNING      0       0       0,0,0

  1      1   SCHED_IDLE         0       1       0,0,0

  2      1   SCHED_IDLE         0       1       0,0,0

  3      1   SCHED_IDLE         0       1       0,0,0

  4      1   SCHED_IDLE         0       1       0,0,0

  5      1   SCHED_IDLE         0       1       0,0,0

  6      1   SCHED_IDLE         0       1       0,0,0

  7      1   SCHED_IDLE         0       1       0,0,0

  8      1   SCHED_IDLE         0       1       0,0,0

end of log:

PANIC: Calling “dbi shutdown” for thread fr_starCLO_cmd

PANIC: Calling “dbi shutdown” for thread fr_starCLO_xlate

PANIC: Calling “dbi shutdown” for thread ib_xp6

PANIC: Calling “dbi shutdown” for thread ob_ishelfadt

PANIC: Calling “dbi shutdown” for thread ob_maclab

PANIC: Calling “dbi shutdown” for thread ob_muse

PANIC: Calling “dbi shutdown” for thread ob_pacs

PANIC: Calling “dbi shutdown” for thread ob_sm

PANIC: Calling “dbi shutdown” for thread ob_sq

PANIC: Calling “dbi shutdown” for thread ob_maclab

03/27/2008 12:58:17

[dbi :dbi :WARN/0:    ob_maclab] NULL DTD when closing DBI

PANIC: Calling “dbi shutdown” for thread ob_maclab

03/27/2008 12:58:17

[dbi :dbi :WARN/0:    ob_maclab] NULL DTD when closing DBI

PANIC: Thread panic—engine going down

PANIC: assertion ‘str != ((char *) 0)’ failed at str.cpp/36

We’re on 5.3 rev3 on Windows 2003.   Everyone is now very concerned that we have a ‘major’ vulnerability on our interface engines.  Can anyone comment on this?

Thank you very much for any insight you may have.

Topic

[Michael Hertel]

I can confirm that I’ve seen this behavior before.

Anything that connects to a process’ command port and sends garbage will cause the process to panic.

I think of this more as a safeguard then a vulnerability.

[Russ Ross]

I have similar experience that running port scans crashed every interface on our cloverleaf server.

At least you knew they were doing a port scan.

I had to spend 3 weeks to catch the guilty system admins that kept saying they weren’t doing anything.

By the way, we run on AIX so this may not just be a problem with Windows.

I keep pointing out the risk of such monitoring to those that only have a vested interest in the monitoring.

Russ Ross
RussRoss318@gmail.com

[Debra Downs]

Thank you both very much.  I appreciate your insight.

[Rob Abbott]

FWIW, this issue is fixed in 5.5 rev2 and 5.6.

Rob Abbott
Cloverleaf Emeritus

[Andre Duguay]

Hi,

(Cloverleaf 5.6 on AIX 5.3)

We just “suffered” the same kind of vulnerability test a few days ago. I can see in our logs that they were scanning numerous ports on the engine including port numbers such as ports 28, 31, 34, 46, 51, 57, 58, 69, etc which I think are internal command ports used by Cloverleaf. We were also scanned on ports where we have inbound threads listening in (7005, 7507, 30201, 30202, 30303, 30525, etc).

The scan on the command ports do not seem to have destabilised the processes but the scans on our inbound threads that were at state opening and the one that were up but configured as multi-server were brought down with pdl errors.

Is that normal? Is there a way within Cloverleaf to protect against something like that?

Thanks,

Andr

[John Mercogliano]

We had the same issue on 5.2. Our compliance group has been salivating since we told them that we are completely up on 5.7 and this problem should be completely fixed.  

We are working on setting up a test scan with all of our interfaces up on our test box to see if everything holds.

I’ll let everyone know how 5.7 holds up.

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Andre Duguay]

John,

According to Rob Abbott in a previous message in that same thread, there should not be a problem with 5.6 and above but we are on 5.6 and still several of our connections went down. So good luck with that upcoming test scan.

[Rob Abbott]

Andre, can you be more specific on what you mean by “brought down with PDL errors?”  Perhaps provide a log file snippet as Mike did?

Tech Support will be happy to help you with this if you have any log files left from the test – send to techsupport@healthvision.com

Also note this thread is nearly 2 years old.  Since that time we found a specific issue with a PDL multi-server panic that was resolved in 5.6 rev1.  Are you on 5.6 without a rev?

I’m not aware of any other issues port scanners / penetration tests would cause with PDL.  If you’ve uncovered something we would be very interested in the details.  Again please work with Tech Support.

Thanks

Rob Abbott
Cloverleaf Emeritus

[Andre Duguay]

Hi Rob,

We are on 5.6 rev2.

I was kinda vague indeed in my statement “brought down with PDL errors” … sorry about that. Here is an example of one of our log that day where we had a multiserver connexion go down with the scan (I have stripped away all irrelevant log noise).

The scan was started at 13:37.

Thread “lisin1” goes down at 13:55.

We restart it at 14:01.

I will follow your suggestion and send this to techsupport.

Thanks

[Andre Duguay]

I just realised that I should also mentioned that we were not just hit with a port scan but also with a VA (Vulnerability Assessment) test.

[Rob Abbott]

Thanks Andre!  This info will help a lot.

Rob Abbott
Cloverleaf Emeritus

[Andre Duguay]

Here is the response from techsupport:

---

Hello Andre,

We would recommend putting your Production servers on the exception list from this scan. There are some potential problems that could occur as a result of running port vulnerability checks. Mainly it could cause a port to be locked or occupied when Cloverleaf is trying to use them when communicating with outside vendors. Also we have some past

cases from clients reporting PANICS in the engine as a result of memory or network resources running low while running such scans. We also have some cases from clients reporting monitord crashing as a result of such scan. Unfortunately we do not currently have any on documentation on port scans in Cloverleaf. However we have had some reports of clients

having trouble as a result of port scans.

[John Mercogliano]

Hi all,

  We just had an unannounced port scan from our lovely IT department on 5.7 rev 2.  We did much better, a lot of weird errors in the log but no message loss and it did not appear that any actual connections went down.

Does anyone from Lawson know what imhVerify is?

Here are the errors we recieved on one connection:

[icl :icl :ERR /0:to_4medica_adt_ord:03/16/2011 22:29:33] imhVerify failed at free 0x40c47d90

[icl :icl :ERR /0:to_4medica_adt_ord:03/16/2011 22:29:33] 0x40c47d90 type is 5

[cmd :cmd :ERR /0:4medica_obd_cmd:03/16/2011 22:29:48] Invalid client connection.  Closing connection.

[cmd :cmd :ERR /0:4medica_obd_cmd:03/16/2011 22:29:48] Invalid client connection.  Closing connection.

John Mercogliano
Sentara Healthcare
Hampton Roads, VA

[Author]

Replies