DI lab results connection closing

[Robert Denny]

We recently have started having issues with our DI manager connection being closed by Cloverleaf and then will not reopen until someone stops and restarts the thread.

any suggestions on what might be causing the PORT to close and then not open until manually reset?

Topic

[Jim Kosloskey]

Do you mean the thread goes ‘Dead’ or ‘Opening’. Is the Cloverleaf connection Client or Server?

Is there any indication in the Process Log of anything amiss?

What release of Cloverleaf, what protocol?

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Robert Denny]

Jim,

Our helpdesk has stated that it goes into an Opening Status.

Here is text from the DI Manager log.

Time Stamp                        State     Message

6/7/2015 05:37:42.057    ERROR  Remote-host closed server process : Port 49103 at IP Address shspwim

6/7/2015 05:37:47.632    STATE    TcpClientConnect^imdevio

6/7/2015 05:38:03.367    TCPIP    Unable to connect to remote-host server process : Port 49103 at IP Address shspwim

This continues every 30seconds or so until the interface in IM was stopped.

6/7/2015 06:39:50.156     STATE     stopped

6/7/2015 06:41:42.061     STATE     started

6/7/2015 06:41:42.061     STATE     TcpClientConnect^imdevio

6/7/2015 06:41:57.766     TCPIP      Unable to connect to remote-host server process : Port 49103 at IP Address shspwim

Restarting interface in IM didn

[Jim Kosloskey]

Well is the port that is being used/configured really 49103 or is that the ‘assigned’ port as part of the connection start?

The reason I ask is that port is in the normal ephemeral range and that could be an issue. Typically we don’t define ports in the ephemeral range.

Also is there a firewall involved? If so perhaps there is a timeout or some other condition which caused the connection to be reset.

Do you have anything on the Cloverleaf side which automatically cycles the connection (like an Alert which stops/starts the thread)? If so find out when that occurred and check the log around then.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Robert Denny]

That is the port that we have used since the process was implemented. The system is configured to send the results via that port and Cloverleaf is listening on it. I do not clearly understand what the ephemeral range really relates too, I will do some reading on it. Changing the port number may resolve the issue, but we have been using the process for around 3 years now.

Also, it seems to be happening every 48 hours on the minute. At 5:55am the connection goes down. The DI system sends messages and waits for an ACK, which is never received and the PORT closes on Cloverleaf and never opens back up until the thread is stopped and restarted on Cloverleaf.

[Kevin Crist]

We had many ports in the ephemeral range and never had an issue until we went from Unix to Linux but i had to change over a hundred ports so they were not in that range. Not fun. Not saying that is the issue but it’s one worth trying if your at a loss.

[Terry Kellum]

It sounds like the connection is being closed by one end of the conversation while the other side never receives the RST packet.  If you know specifically exactly when this started, check with your network folks in both organizations to determine what changes were made to network equipment and configurations during the time that this started.

You might also consider turning on keepalive packets on these connections.  

http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html

http://www.golinuxhub.com/2013/03/setting-up-custom-tcpip-keep-alive.html

Setting up the system for keepalives is one side of the issue.  The low level connection details must be set up to invoke keepalives.  That would probably be a question for Rob Abbott.

[Author]

Replies