Clovertech

Intermittent process panic after cycling thread

› Clovertech Forums › Read Only Archives › Cloverleaf › Cloverleaf › Intermittent process panic after cycling thread

  • Creator
    Topic
  • June 3, 2010 at 6:37 pm #51789
    David Pace
    Participant

      We are on v5.4.1 running on windows server 2003.

      About a month ago, in response to an intermittent problem receiving messages from Cerner, I added alerts on two inbound threads.  I set them to stop and restart if no messages had been received for 30 minutes.

      using the commands:

      hcicmd -p -c ‘ -pstop’

      hcicmd -p -c ‘ -pstart’

      These are called by the alert and processed via a batch file.

      As soon as I did this, I started have intermittent engine panics about weekly.  This is an extract from the log last night:

      [cmd :cmd :INFO/0:       p3_cmd:06/02/2010 20:46:15] Receiving a command

      [cmd :cmd :INFO/0:       p3_cmd:06/02/2010 20:46:15] Received command: ‘cer_dft_in pstart’

      [cmd :cmd :INFO/0:       p3_cmd:06/02/2010 20:46:15] Doing ‘pstart’ command on ‘cer_dft_in’

      [pti :sign:WARN/0:   cer_dft_in:06/02/2010 20:46:16] Thread 11 received signal EXCEPTION_ACCESS_VIOLATION:

       The thread attempted to read from or write to a virtual address for which it does not have the appropriate access.

      [pti :sign:WARN/0:   cer_dft_in:06/02/2010 20:46:16] PC = 0xffffffff

      PANIC: “0”

      PANIC: Calling “pti” for thread p3_cmd

      I have to eventually reboot the server to get everything running again.  So my questions are:

      1) What could be causing the intermittent panics and what do I do to prevent them?

      2) Is cycling an inbound thread this way the best way to keep transactions coming in if there are extended periods with no transactions on the ib thread?

      3) Short of rebooting the server, how can I get the processes and threads up and running again?

      Thanks in advance,

      David

    • Creator
      Topic
    Viewing 1 reply thread
    • Author
      Replies
      • June 4, 2010 at 12:35 pm #71758
        David Harrison
        Participant

          I can only attempt to answer question 2. Why should you need to recycle the threads?

          If Cloverleaf is listening for an inbound connection then it is the responsibility of the sending system to connect as required. In case of the problems you describe with messages not arriving for a period of time, it may be worth using netstat to check the socket state on the sending system.

          If Cloverleaf is making the connection then you can set it to auto reconnect if the connection is lost.

        • June 4, 2010 at 4:59 pm #71759
          John Mercogliano
          Participant

            David,

             You could be running into tcp port exhaustion as an issue.  Windows 2003 has a small ephemeral port range of 1024 thru 5000 unless you have changed it.  If you do a netstat an see a lot of fin_wait_2 states, those will be windows waiting for your previous cerner connections to close down.  Usually, the wait is two hours before windows will release the port back to the pool.  

              Do you have any other dynamic port apps running on the server(ftp, biztalk, web)  or do you have a lot of threads?

              That really is the only thing that I can think of on a windows machine that could lead to that situations just from bounce a thread.

              But as David H. said, you probably need to hit up cerner again.  

            Hope this helps.

            John Mercogliano
            Sentara Healthcare
            Hampton Roads, VA

        • Author
          Replies
        Viewing 1 reply thread
        • The forum ‘Cloverleaf’ is closed to new topics and replies.