Clovertech

Athenahealth, TCP/IP, and DMZ’d SSH box to Cloverleaf

› Clovertech Forums › Read Only Archives › Cloverleaf › Cloverleaf › Athenahealth, TCP/IP, and DMZ’d SSH box to Cloverleaf

  • Creator
    Topic
  • April 27, 2011 at 9:47 pm #52439
    Robbie Parker
    Participant

      I wanted to see if anyone has any experience with Athenahealth communicating with a VanDyke SSH box in a DMZ, which then connects to Cloverleaf- and if so, or similar, could lead me down the proper path of who to send “the” emails to 🙂

      …and this is long.   Forgive me!

      We have had this route implemented since 2007.  Athenahealth (hosted in Mass.) connects to a box in our DMZ that houses VanDyke VShell SSH app.  This DMZ box is then the client and server Cloverleaf connects to for routing to/from one of our EMRs.  It had been very stable, and still is, except for one nuance.

      It is known that when first mgs triggered for the day from Athena app by ambulatory office users, it takes a good 1 to 3 minutes for connections to be made and data ultimately routed and delivered via Cloverleaf.  After that inital front, all things are fairly instantaneous.  Our offices open at 8am, and I have noticed in past audits that first msg hits the thread at 8:02am.  (I just never paid attention to the timestamp in the msg).  Still fairly normal at this point considering the ramp up.

      User opened ticket about some delays a few days ago.  Apparently, one of the offices has staff come in at 7:30-ish and triggers data b/t then and 8am, but *still* seeing data not hit thread to route until 8:02am.

      Opened ticket with Athena for this travesty and sure enough their log says winsock error, must be on my side.  Even recommended to make sure the receiving system was on 🙂

      So, looked at VShell logs and sure enough.  Below, from today:

      07:58:15,fwd ,00011: Port forwarding connection to :9960 failed.  Winsock error: The remote system refused the connection.  This probably means that the remote system does not provide the service you are attempting to access, or that the service is being provided on a different port.

      Now that said, I am fine with that other than WHY at 8:02am all of a sudden everyday it finally connects.  This makes me think they are doing something different at that time.  The box has had all port forwarding wide open since 2007.  I keep 180 days of logs on cloverleaf and this had been ongoing for at least half a year, just that users just made us aware.  We are only talking about a few msgs here and there, but it is the principle I suppose.

      Anyone have any experience or words of wisdon for this?  I am thinking their ports are off until 8:02am and that I need to get netadmin to get me some packets to show them (???)

      *This has to be one of the longest rambles on Clovertech*

      Thanks in advance for any assistance.

    • Creator
      Topic
    Viewing 8 reply threads
    • Author
      Replies
      • April 28, 2011 at 1:06 pm #74200
        Rob Abbott
        Keymaster

          I would set up a sniffer both inside and outside the DMZ to see if the SYN (the packet that initiates a connection) is actually leaving the VanDyke box and making it to the Cloverleaf box.  

          It may be a firewall rule that is preventing the connection outside of business hours?

          Rob Abbott
          Cloverleaf Emeritus

        • April 28, 2011 at 2:05 pm #74201
          Robbie Parker
          Participant

            Thanks Rob.  Netadmin claims he is doing nothing but I will get him to confirm again.

            Say there is no firewall limitations occurring, would there be a next step of troubleshooting.

            Netadmins are ornery people here 🙂

          • April 28, 2011 at 9:08 pm #74202
            Rob Abbott
            Keymaster

              Run the sniffer (wireshark is very good) and see what’s going on at the network level.  

              If you see a SYN leave VanDyke and not make it to CL, then something is blocking or dropping that packet in between.

              If the packet is making it through, then look at what is returned from the CL machine.  I think a RST is returned if nobody’s listening on that port.

              If you see a SYN ACK come back, then CL has accepted the connection.

              The other thing you can do is run “netstat -an” on the CL machine to verify that there is something listening on the port in question.

              Rob Abbott
              Cloverleaf Emeritus

            • May 2, 2011 at 9:27 pm #74203
              Robert Milfajt
              Participant

                If you’re running on AIX, the iptrace utility is quite useful for detecting these types of things.  It’s like a sniffer, but its AIX, so you don’t need to track down your network guy who says the sniffer buffer only holds x amount of data (much less than what you need), and you can run it yourself, given you have root access to your box.

                Hope this helps,

                Robert Milfajt
                Northwestern Medicine
                Chicago, IL

              • May 3, 2011 at 3:15 pm #74204
                Robbie Parker
                Participant

                  We are running 5.6Rev2 on Windows 2003.

                  ..having now acquired Wireshark user status  ðŸ™‚

                  I had Wireshark capture from 7am to 8:30am today.  I have matched the timestamp from the VShell log of:

                  07:48:16,fwd ,00007: Port forwarding connection to :9960 failed.  Winsock error: The remote system refused the connection.  This probably means that the remote system does not provide the service you are attempting to access, or that the service is being provided on a different port.

                  with the SSH to Clovereleaf comm packets in the capture, and I see the SYN and RST,ACK:

                  from VShell to Cloverleaf>

                  No.     Time        Source                Destination           Protocol Info

                     434 3093.046911                     TCP      altav-remmgt > 9960 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 SACK_PERM=1

                  from Cloverleaf to VShell>

                  No.     Time        Source                Destination           Protocol Info

                     435 3093.047678                     TCP      9960 > altav-remmgt [RST, ACK] Seq=4029370377 Ack=1 Win=0 Len=0

                  this is what Rob mentioned to look for and that the RST means Cloverleaf is not listening…but it is (???)

                  At 8:02:15 (time when the first connection is made and data delivered the entries are:

                  from VShell to Cloverleaf>

                  No.     Time        Source                Destination           Protocol Info

                     849 3931.962964                     TCP      direcpc-si > 9960 [PSH, ACK] Seq=4506 Ack=769 Win=64767 Len=573

                  from Cloverleaf to VShell>

                  No.     Time        Source                Destination           Protocol Info

                     850 3931.971360                     TCP      9960 > direcpc-si [PSH, ACK] Seq=769 Ack=5079 Win=32768 Len=96

                  After this point, connection is good throughout the day.  I guess my question now is, is there something on Cloverleaf I can change, or do I have enough info to ‘engage the vendor with a slight lean toward “what the heck are you doing at 8:02am that you are not doing before then?” ‘

                  Think the direcpc-si and the altav-remmgt are Athenahealth hostnames and that may have something to do with it?

                  I enjoy the logic/coding portion of this job- not the networking 🙂

                • May 3, 2011 at 6:06 pm #74205
                  Rob Abbott
                  Keymaster

                    Quote:

                    this is what Rob mentioned to look for and that the RST means Cloverleaf is not listening…but it is (???)

                    Are you sure CL is listening?  Or does CL think the session is already established?  What does netstat say about this port on the CL box at 8AM?

                    If the connection has been idle for some time the firewall may have torn down the connection without notifying either end.  When the remote end tries to establish a new connection, it’s refused because CL thinks it’s already connected.

                    After the RST/ACK do you see another connection being established (SYN, SYN/ACK, ACK)?  If not, then there is an already established session that’s being used.

                    If you turn on multiserver, CL will be able to accept multiple connections on the same port which may resolve this.

                    Rob Abbott
                    Cloverleaf Emeritus

                  • May 4, 2011 at 2:05 pm #74206
                    Robbie Parker
                    Participant

                      Rob,

                      I really appreciate your assistance here.  I think you may be correct regarding the firewall.

                      I captured packets from SSH box and Cloverleaf box this morning, though have yet to compare side by side..

                      I did run netstat at 7:54am and 7:59am and the connection is ESTABLISHED on the port / thread in question (thread status=Up at this time as well).  The interesting part is at 8:02, I saw the ib Athena thread go from Up to Opening, then Up.  At that time that the connection is made for delivery from SSH box, and data is passed…  

                      Is the multiserver option a quick fix from the cloverleaf side and somewhat normal protocol for troublesome threads or if it were you would you prod the Firewall chief?  ðŸ™‚

                      Thanks again.

                    • May 5, 2011 at 3:26 pm #74207
                      Robbie Parker
                      Participant

                        Yesterday, I changed the thread from server to multiserver and set to “2” and immediately caused thread errors, so I removed that.

                        Thinking along the same lines that Athena, or Firewall, or VShell has broken connection, but CL thread still thinks connected that Rob mentioned, I scripted cycling the process owning the thread in question today at 7am and 7:30am.

                        I was hoping then this would make sure all parties then aware ‘pipe broken, rebuild’- it seemed to work today.  Pre 8am msgs were delivered in an appropriate timeframe.

                        I hope this continues to be the case over the next few days.

                        Even though the exact reason is unknown to me, this simple, acceptable work around will do what I need if indeed it keeps giving good results.

                        I thank Rob and Robert for their feedback.  Cheers!

                      • May 11, 2011 at 6:22 pm #74208
                        Robbie Parker
                        Participant

                          Cycling the owning process pre 8am to force CL to close the socket has done the trick here.  I have had successful, timely transfers for 7 business days at this time.

                          Thanks again for the assistance.

                      • Author
                        Replies
                      Viewing 8 reply threads
                      • The forum ‘Cloverleaf’ is closed to new topics and replies.