Network Scan causes engine panic.

Homepage Clovertech Forums Read Only Archives Cloverleaf Cloverleaf Network Scan causes engine panic.

  • Creator
    Topic
  • #49944
    Debra Downs
    Participant

    Last Thursday our compliance group had an outside company performing a vulnerability scan on our network.  The scan was at the ‘port scanning’ stage when all three of our engines panic’d and went down:

    [cmd :cmd :INFO/0:fr_starCLO_cmd] Receiving a command

    PANIC: “str != ((char *) 0)”

    PANIC: Calling “pti” for thread fr_starCLO_cmd


    Scheduler State


    Thread Events     State      Priority Runnable  PT Msgs

      0      0   SCHED_RUNNING      0       0       0,0,0

      1      1   SCHED_IDLE         0       1       0,0,0

      2      1   SCHED_IDLE         0       1       0,0,0

      3      1   SCHED_IDLE         0       1       0,0,0

      4      1   SCHED_IDLE         0       1       0,0,0

      5      1   SCHED_IDLE         0       1       0,0,0

      6      1   SCHED_IDLE         0       1       0,0,0

      7      1   SCHED_IDLE         0       1       0,0,0

      8      1   SCHED_IDLE         0       1       0,0,0

    end of log:

    PANIC: Calling “dbi shutdown” for thread fr_starCLO_cmd

    PANIC: Calling “dbi shutdown” for thread fr_starCLO_xlate

    PANIC: Calling “dbi shutdown” for thread ib_xp6

    PANIC: Calling “dbi shutdown” for thread ob_ishelfadt

    PANIC: Calling “dbi shutdown” for thread ob_maclab

    PANIC: Calling “dbi shutdown” for thread ob_muse

    PANIC: Calling “dbi shutdown” for thread ob_pacs

    PANIC: Calling “dbi shutdown” for thread ob_sm

    PANIC: Calling “dbi shutdown” for thread ob_sq

    PANIC: Calling “dbi shutdown” for thread ob_maclab

    03/27/2008 12:58:17

    [dbi :dbi :WARN/0:    ob_maclab] NULL DTD when closing DBI

    PANIC: Calling “dbi shutdown” for thread ob_maclab

    03/27/2008 12:58:17

    [dbi :dbi :WARN/0:    ob_maclab] NULL DTD when closing DBI

    PANIC: Thread panic—engine going down

    PANIC: assertion ‘str != ((char *) 0)’ failed at str.cpp/36

    We’re on 5.3 rev3 on Windows 2003.   Everyone is now very concerned that we have a ‘major’ vulnerability on our interface engines.  Can anyone comment on this?

    Thank you very much for any insight you may have.

Viewing 12 reply threads
  • Author
    Replies
    • #64202
      Michael Hertel
      Participant

      I can confirm that I’ve seen this behavior before.

      Anything that connects to a process’ command port and sends garbage will cause the process to panic.

      I think of this more as a safeguard then a vulnerability.

    • #64203
      Russ Ross
      Participant

      I have similar experience that running port scans crashed every interface on our cloverleaf server.

      At least you knew they were doing a port scan.

      I had to spend 3 weeks to catch the guilty system admins that kept saying they weren’t doing anything.

      By the way, we run on AIX so this may not just be a problem with Windows.

      I keep pointing out the risk of such monitoring to those that only have a vested interest in the monitoring.

      Russ Ross
      RussRoss318@gmail.com

    • #64204
      Debra Downs
      Participant

      Thank you both very much.  I appreciate your insight.

    • #64205
      Rob Abbott
      Keymaster

      FWIW, this issue is fixed in 5.5 rev2 and 5.6.

      Rob Abbott
      Director, Product Management - Infor Cloverleaf

    • #64206
      Andre Duguay
      Participant

      Hi,

      (Cloverleaf 5.6 on AIX 5.3)

      We just “suffered” the same kind of vulnerability test a few days ago. I can see in our logs that they were scanning numerous ports on the engine including port numbers such as ports 28, 31, 34, 46, 51, 57, 58, 69, etc which I think are internal command ports used by Cloverleaf. We were also scanned on ports where we have inbound threads listening in (7005, 7507, 30201, 30202, 30303, 30525, etc).

      The scan on the command ports do not seem to have destabilised the processes but the scans on our inbound threads that were at state opening and the one that were up but configured as multi-server were brought down with pdl errors.

      Is that normal? Is there a way within Cloverleaf to protect against something like that?

      Thanks,

      Andr

    • #64207
      John Mercogliano
      Participant

      We had the same issue on 5.2. Our compliance group has been salivating since we told them that we are completely up on 5.7 and this problem should be completely fixed.  

      We are working on setting up a test scan with all of our interfaces up on our test box to see if everything holds.

      I’ll let everyone know how 5.7 holds up.

      John Mercogliano
      Sentara Healthcare
      Hampton Roads, VA

    • #64208
      Andre Duguay
      Participant

      John,

      According to Rob Abbott in a previous message in that same thread, there should not be a problem with 5.6 and above but we are on 5.6 and still several of our connections went down. So good luck with that upcoming test scan.

    • #64209
      Rob Abbott
      Keymaster

      Andre, can you be more specific on what you mean by “brought down with PDL errors?”  Perhaps provide a log file snippet as Mike did?

      Tech Support will be happy to help you with this if you have any log files left from the test – send to techsupport@healthvision.com

      Also note this thread is nearly 2 years old.  Since that time we found a specific issue with a PDL multi-server panic that was resolved in 5.6 rev1.  Are you on 5.6 without a rev?

      I’m not aware of any other issues port scanners / penetration tests would cause with PDL.  If you’ve uncovered something we would be very interested in the details.  Again please work with Tech Support.

      Thanks

      Rob Abbott
      Director, Product Management - Infor Cloverleaf

    • #64210
      Andre Duguay
      Participant

      Hi Rob,

      We are on 5.6 rev2.

      I was kinda vague indeed in my statement “brought down with PDL errors” … sorry about that. Here is an example of one of our log that day where we had a multiserver connexion go down with the scan (I have stripped away all irrelevant log noise).

      The scan was started at 13:37.

      Thread “lisin1” goes down at 13:55.

      We restart it at 14:01.

      I will follow your suggestion and send this to techsupport.

      Thanks

    • #64211
      Andre Duguay
      Participant

      I just realised that I should also mentioned that we were not just hit with a port scan but also with a VA (Vulnerability Assessment) test.

    • #64212
      Rob Abbott
      Keymaster

      Thanks Andre!  This info will help a lot.

      Rob Abbott
      Director, Product Management - Infor Cloverleaf

    • #64213
      Andre Duguay
      Participant

      Here is the response from techsupport:


      Hello Andre,

      We would recommend putting your Production servers on the exception list from this scan. There are some potential problems that could occur as a result of running port vulnerability checks. Mainly it could cause a port to be locked or occupied when Cloverleaf is trying to use them when communicating with outside vendors. Also we have some past

      cases from clients reporting PANICS in the engine as a result of memory or network resources running low while running such scans. We also have some cases from clients reporting monitord crashing as a result of such scan. Unfortunately we do not currently have any on documentation on port scans in Cloverleaf. However we have had some reports of clients

      having trouble as a result of port scans.

    • #64214
      John Mercogliano
      Participant

      Hi all,

        We just had an unannounced port scan from our lovely IT department on 5.7 rev 2.  We did much better, a lot of weird errors in the log but no message loss and it did not appear that any actual connections went down.

      Does anyone from Lawson know what imhVerify is?

      Here are the errors we recieved on one connection:

      [icl :icl :ERR /0:to_4medica_adt_ord:03/16/2011 22:29:33] imhVerify failed at free 0x40c47d90

      [icl :icl :ERR /0:to_4medica_adt_ord:03/16/2011 22:29:33] 0x40c47d90 type is 5

      [cmd :cmd :ERR /0:4medica_obd_cmd:03/16/2011 22:29:48] Invalid client connection.  Closing connection.

      [cmd :cmd :ERR /0:4medica_obd_cmd:03/16/2011 22:29:48] Invalid client connection.  Closing connection.

      John Mercogliano
      Sentara Healthcare
      Hampton Roads, VA

Viewing 12 reply threads
  • The forum ‘Cloverleaf’ is closed to new topics and replies.

Forum Statistics

Registered Users
5,053
Forums
28
Topics
9,236
Replies
34,175
Topic Tags
273