Port hangs

[Jim]

Cloverleaf version 6.1 on RHEL 6

We have been experiencing port hangs on multiple threads we have rebooted our Linux system and the receiving systems have also rebooted their systems with no resolution.

When we change to a different port and bring the systems back up everything connects and the messages are processed.

We have also v-motioned our system to a different chassis again with no resolution.

I have opened a case with McKesson but I hoped somebody else has experienced the same type of issue.

Does anybody have any insight they would like to share?

Thank you,

Jim

Topic

[Keith McLeod]

I did have something similar and ultimately I believe it was an open files issue.

[Russ Ross]

Sounds like 2 interfaces trying to connect to the same port.

The last time I had similar circumstances I used TCP/IP trace to determine there was an unintended server connecting in addition to the live server.

My analysis eventually revealed the disaster recovery interfaces on some of the servers had been turned on unknowingly.

Once those DR interfaces were turned off then problem solved.

Changing port numbers is an immediate work around until the port conflicts can be identified.

Coming to grips with port conflicts and how to manage them is likely the bigger issue and the hung ports might be symptoms.

Russ Ross
RussRoss318@gmail.com

[Elisha Gould]

A couple of things that I have seen:

If there is a long period between messages, the networking equipment may timeout the connection. The result will be the server does not know about the disconnection and holds the port open.

Another possible option if its something connecting to the wrong environment, is to use different port ranges for the different environments.

The easy way to check for the above cases is to change to multi-server and monitor for multiple connections.

Also never use ports at or above 32768 in *nix, as these are ephemeral ports.

[Russ Ross]

Good point about not using ports in the ephemeral range (on our box anything above 32K).

Interfaces configured to use ports in the ephemeral range is just a matter of time before they malfunction and can even impact the cloverleaf host server and hang that too.

Definitely make sure none of the ports are in the ephemeral range.

The ephemeral range can be configured to be a different range on a given server.

If unable to figure out how to display what your server is set too, start with getting rid of any ports above 32K = 1024×32 = 32768.

We learned a long time ago not to use ephemeral range ports that I had almost forgotten about that, plus our port management scripts prevent that from happening anymore.

Russ Ross
RussRoss318@gmail.com

[Author]

Replies