Another VPN/Firewall Vendor Battle

[Lori Owen]

We have 2 inbound threads from a vendor over a VPN. It ‘appears’ that the vendor is not releasing our port and we have had to bounce the threads to open the ports back up. Before we bounce, we have done a netstat -a and the ports are not listed – they are gone. Is this normal behavior? We bounce the threads and the ports are back in a LISTENING status. We have gone back and forth with the vendor and they keep blaming Cloverleaf. We have put in stop/start jobs (that worked) and have finally set the ports to multi-server which has helped as well. The vendor’s network guy sent the below info after we “suggested” that the ports were not getting released after the VPN closed. Any thoughts?

“It is a system/software limitation and not a function of the VPN tunnel itself.  The VPN tunnel is just a conduit to route traffic from our network to yours via an internet connection.

The opening/closing of TCP sessions to and from a port would be the function of the software that is running on either side of the VPN tunnel.  When a message is transmitted and the connection closed the VPN tunnel is just passing that traffic and not actually involved in the process.  While we were looking at things and having problems earlier I did not see the actual VPN tunnel go down.”

Thanks!

Topic

[Carson Nguyen]

Lori Owen wrote:

We have 2 inbound threads from a vendor over a VPN. It ‘appears’ that the vendor is not releasing our port and we have had to bounce the threads to open the ports back up. Before we bounce, we have done a netstat -a and the ports are not listed – they are gone. Is this normal behavior? We bounce the threads and the ports are back in a LISTENING status. We have gone back and forth with the vendor and they keep blaming Cloverleaf. We have put in stop/start jobs (that worked) and have finally set the ports to multi-server which has helped as well. The vendor’s network guy sent the below info after we “suggested” that the ports were not getting released after the VPN closed. Any thoughts?

“It is a system/software limitation and not a function of the VPN tunnel itself.

[Elisha Gould]

As what Carson said.

We often have issues with firewalls/VPN’s not sending the disconnect notification after a timeout. Its due to different TTL’s being used and packets being dropped.

Since cloverleaf does not have the option to disconnect a client after not receiving a message for a period of time, the use of a multi server is the easiest method.

If there should only be one connection, its a good idea to have a tclproc on the ib TPS that checks the client_list global variable for connections and send a PROTO message to close the other connections.

This closes off the now defunct connections when the first message is recieved.

You can leave them and they will eventually get cleaned up (depending on the TTL settings), but if it occurs too often it can cause all of the available ports to be used up.

Below is some sample code to grab the defunct connections and close off the connections.

You’ll need multi server and “Use DRIVERCTL control” enabled.

Code:

global client_list

.
.
.

set myReturn “CONTINUE $aMsgId”

set myConnId “”
set myDriverCtrl [msgmetaget $aMsgId DRIVERCTL]
keylget myDriverCtrl CONNID myConnId

set myCloseList [lsearch -inline -all -not -exact $client_list $myConnId]

foreach myClient $myCloseList  {
   set myMsgId [msgcreate]
   set myDriverCtl {}
   keylset myDriverCtl CLOSE 1
   keylset myDriverCtl WRITEZERO 0
   keylset myDriverCtl CONNID $myClient
   msgmetaset $myMsgId DRIVERCTL $myDriverCtl
   lappend myReturn “PROTO $myMsgId”
}

[/code]

[Lori Owen]

Thanks for the feedback. In the meantime, we set the thread back to ‘Server’ and checked the ‘Close after write’ box. So far we have maintained a consistent connection. Does this support the theory that the vendor is not sending a disconnect? Should the VPN be sending the disconnect or the actual application/service sending the messages?

Thanks,

Lori

[Elisha Gould]

This is the how firewalls/vpn/networks work by design to stop unwanted traffic. There isn’t anything wrong with the vendor or VPN.

They likely have different TTL’s, so your connection thinks its ok, but someware else doesn’t. When this happens no more packets will be passed through.

The server will eventually time out the connection and attempt to send a packet to see if the connection is still open, but wont get a responce, so it will close off the connection. This will take however long the TTL is set to.

So if the server’s TTL is 24 hours and the firewall/VPN is 1 hour, the firewall/VPN will start blocking traffic after 1 hour of inactivity, but the server will wait another 23 hours before it realises the connection is down.

Be a little careful with the ‘Close after write’, as it will take some time to come back up again. Its ok if your throughput is 1 message/minute or less, but I wouldn’t use it for any higher rate.

[Robert Kersemakers]

Also have a look at your TCP keepalive setting.

Normally firewalls will disconnect a tcp/ip connection after a certain time of inactivity; say an hour. This is normal/basic security. By setting the keepalive lower than the timeout setting of the firewall (say half an hour), the server will send an empty packet through tcp/ip after a half hour inactivity. The firewall will see this packet and reset its inactivity counter.

May be a solution as well.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[gricelda lopez]

We currently have the same issue since we moved from egate to clover. The incoming thread would lose connection due to lack of activity. We had to manually stop and restart the thread in order to re-connect and receive messages. I asked the network people to check  TCP keepalive setting and they said they matched the vendors. I wasn’t sure what else to do or what to ask so I created an alert in clover. The alert will call a command to restart the thread if there is no activity > than 10 minutes. So far so good…

[Author]

Replies