TLS problem

Clovertech Forums Cloverleaf TLS problem

  • Creator
    Topic
  • #120037
    David Barr
    Participant

      We’re having a TLS issue between Epic and Cloverleaf. We have potential workarounds for the issue, but none of them are great. This affects standard inbound HL7 TLS interfaces using MLLP (as opposed to HTTPS or something like that).

      When we start an outbound Epic interface, it opens the TCP connection to Cloverleaf, but it doesn’t start the TLS handshake until it has a message to send. This could potentially be hours later (days later in a test environment). When an inbound Cloverleaf interface receives a TCP connection, there is a 30 second timeout for the TLS handshake to occur, after that the connection is reset. During those 30 seconds all other interfaces in the process are non-responsive. Because Epic will restart the connection when it is reset, that means that all Cloverleaf interfaces in the same process as the problematic interface will not work until Epic has a message to send.

      Cloverleaf development says that the issue is a “rare case” and we have workarounds, so they haven’t offered a fix of the non-responsiveness problem. Their suggested workaround is to put each inbound TLS interface in its own process.

      Epic says that Cloverleaf shouldn’t have any expectation that the handshake will happen right when the TCP connection is opened. They’ve suggested that we make the Cloverleaf inbound threads be the clients when setting up the connection.

      Other options I’ve considered are to disable TLS, use HTTPS instead of TLS or leave it the way it is and manually turn off interfaces if they don’t have queued messages.

      I’m posting here as a warning to others who are setting up these interfaces. It took a long time to realize what was happening, and it just seemed like the interfaces were sometimes unreliable.

       

    Viewing 10 reply threads
    • Author
      Replies
      • #120040
        Jim Kosloskey
        Participant

          Thanks, David, for the heads up.

          This should save others time.

          I have my opinion on the matter as I am sure others do but this is valuable information making us forewarned.

          What release of Cloverleaf?

          email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

        • #120041
          David Barr
          Participant

            Cloverleaf 20.1.1.3.

             

          • #120042
            Robert Kersemakers
            Participant

              Thanks for the heads up, David.

              I’m no expert in this, but googling around learns that “TLS handshakes occur after a TCP connection has been opened via a TCP handshake.”
              Big question is whether TLS handshake occurs immediately after TCP handshake. Normally I would say yes: it is weird to open up a TCP connection and then wait with setting up TLS until a message arrives. Why wait for that and add extra delay to that first message when you could have established TLS a long while ago.

              So imho this should be solved by Epic: do the TLS handshake immediately after TCP handshake.

              Also looked whether the TLS timeout can be shortened as another workaround, but can’t find this in Cloverleaf.

              Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

            • #120043
              David Barr
              Participant

                I don’t think shortening the timeout would help unless we could also add a connection retry delay to the Epic side of the interface. Otherwise we would still be stuck in a repeating loop. I can check into that. Epic doesn’t think they are doing anything incorrectly, so they don’t want to spend time fixing it.

              • #120044
                David Barr
                Participant

                  This is from the TLS specification:

                  TLS is application protocol independent; higher-level protocols can layer on top of TLS transparently. The TLS standard, however, does not specify how protocols add security with TLS; how to initiate TLS handshaking and how to interpret the authentication certificates exchanged are left to the judgment of the designers and implementors of protocols that run on top of TLS.

                   

                • #120048
                  Robert Kersemakers
                  Participant

                    Hmm. But still not logical to open a TCP connection and then wait for the first message to do the TLS handshake.

                    Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

                  • #120054
                    Rob Abbott
                    Keymaster

                      Thanks for raising this David – I’ve asked R&D to take a second look and see if there’s a way we can prevent blocking in this scenario.

                      Are you using pdl or the native tcpip driver?   The tcpip driver may behave a little better as it’s multithreaded under the covers.

                      Rob Abbott
                      Cloverleaf Emeritus

                    • #120056
                      David Barr
                      Participant

                        Thanks for looking into this. I’m using the PDL driver. I can test the TCP/IP.

                      • #120058
                        David Barr
                        Participant

                          It looks like the native tcpip driver is working better. I’ll have to test it longer to see if that totally fixes the issue. Thanks again.

                        • #120059
                          Robert Kersemakers
                          Participant

                            Good to hear!

                            Note to self: have a look whether we should/could replace pdl-driver with native tcpip driver. Right now we are still using pdl driver a lot.

                            Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

                          • #120066
                            Rob Abbott
                            Keymaster

                              It looks like the native tcpip driver is working better. I’ll have to test it longer to see if that totally fixes the issue. Thanks again.

                               

                              You are welcome!   Please let us know the results of further testing.  Thanks David!

                              Rob Abbott
                              Cloverleaf Emeritus

                          Viewing 10 reply threads
                          • You must be logged in to reply to this topic.