LARGE VOLUME OF PENDING MESSAGES

Clovertech Forums Cloverleaf LARGE VOLUME OF PENDING MESSAGES

  • Creator
    Topic
  • #115747
    Stewart
    Participant

      We have a vitals feed coming from 2 different vendors into cloverleaf and then being sent to the EMR.  There continues to be a large amount of pending messages in the OB thread to the EMR.  Is this most likely due to the EMR not being able to process the incoming messages fast enough or is it due to a limit with Cloverleaf?

    Viewing 4 reply threads
    • Author
      Replies
      • #115748
        Tipu Razaq
        Participant

          How many messages are coming inbound from the two feeds? Are the inbound/outbound feeds in different processes? We have ~1000 Philips devices here, sending a message every 5 seconds. We filter using a TCL script to only accept every 6th message which may help.

          We’ve ran into instances where a bolus of inbound messages will cause the outbound thread to not send messages out, but that usually clears up once the inbound queue clears. If there are too many messages on the inbound thread, you may just have to delete them from the recovery database of the site that’s hosting the thread so the outbound thread can get going. Cloverleaf seems to dedicate resources where it’s needed and the inbound feed will consume those resources if there are many messages coming through.

          Our Cloverleaf instance is running on Linux if that matters at all.

        • #115749
          Keith McLeod
          Participant

            If you are receiving an ACK back from the EMR, then the EMR is regulating the flow of messages.  This can be caused by how high the debug level is set on your EMR as one possibility.  Generally cloverleaf has no problems keeping up with other interfaces, however, another possibility is the handling of the ACK on cloverleaf side.  I have experienced on McKesson STAR a while back where the ACK message the engine sent back did not match the phrase expected on their side, so they would wait for the timeout before sending the next message.  This gave me a message flow of approx. 1 message every 2 seconds.  Both sides claiming to be configured correctly.  Only this one feed was painfully slow.  Finally got a programmer involved who said I was missing a carriage return at the end of the ACK message for this particular interface. All others didn’t require it.  They in fact needed 3 CR’s and it broke the dam….

          • #115750
            Peter Heggie
            Participant

              We are looking at building a results feed from a vital signs system, and although it is not built yet, we are thinking that we will not to send every result, mostly because our EMR cannot handle such high volume. The EMR is not built to be a continuous feed display/capture tool. All of the results sent into it are stored in the patients clinical record.

              I have not put much thought into it yet but am thinking about building a little piece of code that only passes a subset of readings, perhaps every 5 minutes or more. We need clinical staff buy-in for that methodology, because we don’t know if important outlier data could be discarded if they only presented in-between the interval we use. This could be a show-stopper for the feed.

              As far as your original question, I don’t think the issue is Cloverleaf – it is incredibly fast, even if using the Recovery DB. Perhaps with clinical data, you are not allowed to run without the Recovery DB enabled, but you could ask the question. And since the pending messages are on the OB thread, the likelihood is much greater that the issue is the EMR. I assume the OB thread is tcp/ip, that you are using Auto-Reconnect and not using Close After Write?

              Maybe you could create another interface pair of threads that fits in-between your translation interface and the connection to the EMR, just for the purpose of having another place where the pending messages would build up. It could be in a different process or even a different site, so that a large volume of messages does not impact performance in other processes or sites.

              Peter Heggie

            • #115754
              Vince Angulo
              Participant

                We have Cerner Millennium as our EMR and we experience this type of queuing from time to time.

                Not sure if it’s slow ACKs or the complex server chain that Cerner uses to process inbound traffic.

              • #115759
                David Barr
                Participant

                  What is the message state of the pending messages? If they are all in state 7 (for example), that could indicate a Cloverleaf problem, but if they are in state 11, that’s probably either a network issue or a problem with the EMR.

              Viewing 4 reply threads
              • You must be logged in to reply to this topic.