- This topic has 9 replies, 3 voices, and was last updated 13 years, 4 months ago by
.
-
Topic
-
I wanted to see if anyone has any experience with Athenahealth communicating with a VanDyke SSH box in a DMZ, which then connects to Cloverleaf- and if so, or similar, could lead me down the proper path of who to send “the” emails to 🙂 …and this is long. Forgive me!
We have had this route implemented since 2007. Athenahealth (hosted in Mass.) connects to a box in our DMZ that houses VanDyke VShell SSH app. This DMZ box is then the client and server Cloverleaf connects to for routing to/from one of our EMRs. It had been very stable, and still is, except for one nuance.
It is known that when first mgs triggered for the day from Athena app by ambulatory office users, it takes a good 1 to 3 minutes for connections to be made and data ultimately routed and delivered via Cloverleaf. After that inital front, all things are fairly instantaneous. Our offices open at 8am, and I have noticed in past audits that first msg hits the thread at 8:02am. (I just never paid attention to the timestamp in the msg). Still fairly normal at this point considering the ramp up.
User opened ticket about some delays a few days ago. Apparently, one of the offices has staff come in at 7:30-ish and triggers data b/t then and 8am, but *still* seeing data not hit thread to route until 8:02am.
Opened ticket with Athena for this travesty and sure enough their log says winsock error, must be on my side. Even recommended to make sure the receiving system was on
🙂 So, looked at VShell logs and sure enough. Below, from today:
07:58:15,fwd ,00011: Port forwarding connection to
:9960 failed. Winsock error: The remote system refused the connection. This probably means that the remote system does not provide the service you are attempting to access, or that the service is being provided on a different port. Now that said, I am fine with that other than WHY at 8:02am all of a sudden everyday it finally connects. This makes me think they are doing something different at that time. The box has had all port forwarding wide open since 2007. I keep 180 days of logs on cloverleaf and this had been ongoing for at least half a year, just that users just made us aware. We are only talking about a few msgs here and there, but it is the principle I suppose.
Anyone have any experience or words of wisdon for this? I am thinking their ports are off until 8:02am and that I need to get netadmin to get me some packets to show them (???)
*This has to be one of the longest rambles on Clovertech*
Thanks in advance for any assistance.
-
Replies
-
-
I would set up a sniffer both inside and outside the DMZ to see if the SYN (the packet that initiates a connection) is actually leaving the VanDyke box and making it to the Cloverleaf box.
It may be a firewall rule that is preventing the connection outside of business hours?
Rob Abbott
Cloverleaf Emeritus -
-
Run the sniffer (wireshark is very good) and see what’s going on at the network level.
If you see a SYN leave VanDyke and not make it to CL, then something is blocking or dropping that packet in between.
If the packet is making it through, then look at what is returned from the CL machine. I think a RST is returned if nobody’s listening on that port.
If you see a SYN ACK come back, then CL has accepted the connection.
The other thing you can do is run “netstat -an” on the CL machine to verify that there is something listening on the port in question.
Rob Abbott
Cloverleaf Emeritus -
If you’re running on AIX, the iptrace utility is quite useful for detecting these types of things. It’s like a sniffer, but its AIX, so you don’t need to track down your network guy who says the sniffer buffer only holds x amount of data (much less than what you need), and you can run it yourself, given you have root access to your box.
Hope this helps,
Robert Milfajt
Northwestern Medicine
Chicago, IL -
We are running 5.6Rev2 on Windows 2003.
..having now acquired Wireshark user status 🙂
I had Wireshark capture from 7am to 8:30am today. I have matched the timestamp from the VShell log of:
07:48:16,fwd ,00007: Port forwarding connection to :9960 failed. Winsock error: The remote system refused the connection. This probably means that the remote system does not provide the service you are attempting to access, or that the service is being provided on a different port.
with the SSH to Clovereleaf comm packets in the capture, and I see the SYN and RST,ACK:
from VShell to Cloverleaf>
No. Time Source Destination Protocol Info
434 3093.046911 TCP altav-remmgt > 9960 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 SACK_PERM=1
from Cloverleaf to VShell>
No. Time Source Destination Protocol Info
435 3093.047678 TCP 9960 > altav-remmgt [RST, ACK] Seq=4029370377 Ack=1 Win=0 Len=0
this is what Rob mentioned to look for and that the RST means Cloverleaf is not listening…but it is (???)
At 8:02:15 (time when the first connection is made and data delivered the entries are:
from VShell to Cloverleaf>
No. Time Source Destination Protocol Info
849 3931.962964 TCP direcpc-si > 9960 [PSH, ACK] Seq=4506 Ack=769 Win=64767 Len=573
from Cloverleaf to VShell>
No. Time Source Destination Protocol Info
850 3931.971360 TCP 9960 > direcpc-si [PSH, ACK] Seq=769 Ack=5079 Win=32768 Len=96
After this point, connection is good throughout the day. I guess my question now is, is there something on Cloverleaf I can change, or do I have enough info to ‘engage the vendor with a slight lean toward “what the heck are you doing at 8:02am that you are not doing before then?” ‘
Think the
direcpc-si and thealtav-remmgt are Athenahealth hostnames and that may have something to do with it?I enjoy the logic/coding portion of this job- not the networking 🙂
-
Quote:
this is what Rob mentioned to look for and that the RST means Cloverleaf is not listening…but it is (???)
Are you sure CL is listening? Or does CL think the session is already established? What does netstat say about this port on the CL box at 8AM?
If the connection has been idle for some time the firewall may have torn down the connection without notifying either end. When the remote end tries to establish a new connection, it’s refused because CL thinks it’s already connected.
After the RST/ACK do you see another connection being established (SYN, SYN/ACK, ACK)? If not, then there is an already established session that’s being used.
If you turn on multiserver, CL will be able to accept multiple connections on the same port which may resolve this.
Rob Abbott
Cloverleaf Emeritus -
Rob,
I really appreciate your assistance here. I think you may be correct regarding the firewall.
I captured packets from SSH box and Cloverleaf box this morning, though have yet to compare side by side..
I did run netstat at 7:54am and 7:59am and the connection is ESTABLISHED on the port / thread in question (thread status=Up at this time as well). The interesting part is at 8:02, I saw the ib Athena thread go from Up to Opening, then Up. At that time that the connection is made for delivery from SSH box, and data is passed…
Is the multiserver option a quick fix from the cloverleaf side and somewhat normal protocol for troublesome threads or if it were you would you prod the Firewall chief? 🙂
Thanks again.
-
Yesterday, I changed the thread from server to multiserver and set to “2” and immediately caused thread errors, so I removed that.
Thinking along the same lines that Athena, or Firewall, or VShell has broken connection, but CL thread still thinks connected that Rob mentioned, I scripted cycling the process owning the thread in question today at 7am and 7:30am.
I was hoping then this would make sure all parties then aware ‘pipe broken, rebuild’- it seemed to work today. Pre 8am msgs were delivered in an appropriate timeframe.
I hope this continues to be the case over the next few days.
Even though the exact reason is unknown to me, this simple, acceptable work around will do what I need if indeed it keeps giving good results.
I thank Rob and Robert for their feedback. Cheers!
-
-
- The forum ‘Cloverleaf’ is closed to new topics and replies.
