Linux and Cloverleaf (Ephemeral Ports)

When you say multiple sessions of cloverleaf running are you talking about the host server or something else (ex: same monitor daemon, same cloverleaf process).

There are ways for all these to occur but I want to narrow down the scope of your question before I consider trying to respond.

We are running on AIX and frequently in the past expeirenced multiple instances of the cloverleaf host server magically multiplying itself and eventually slowing down or freezing the GUI/IDE.

There were 2 things that I remediated related to our host server issues listed here:

1) host server multiples itself as if it was started over and over but was only started once

2) host server goes away on its own without stopping it

To fix item 1) like you are experiencing, we learned we were using port numbers in the ephemeral range (in our case that is 32K or above).  I think this was the bigesst cause of the host server spawining copies of itself own its own.  Once we made all our ports below 32K this problem went away and haven’t seen it since, so check what your ephemeral range is on your cloverleaf server and make sure no ports being used fall in that range.  You will have to change all the port numbers being used in the ephermal range to completely get rid of this problem.  The occurrence of the problem will slow down as you eliminate more and more of any ephermal range port violations, but will not completely go away until they are all remediated.

To fix item 2) we learned to take a couple of precautionary steps when launching the host server, the biggest was to unset the $DISPLAY before launching it.  I ended up writing a start_hs.ksh script to make it easy to start the host server reliably, which is posted at this URL.

    https://usspvlclovertch2.infor.com/viewtopic.php?t=6634

Sorry to bump an older thread but i have a question. We just moved to CL 6.0.1 on RHEL 6.5. Since then we have seen a handful of threads that show the error “Unable to bind tcp/ip socket: Address already in use”. and these are threads that have been running for a good while. Our port numbers go to 46K while it is mentioned in this thread to not go above 32K. We get ours throught McKesson unfortunately but the said there shouldnt be an issue until you get to 65K. Seeing the issues we are having i am leaning towards them not being right. Anyone have any advice as maybe the best way to proceed.

Thanks.

You said

This is a symptom that evetully manifests itself in a random fashion when interffaces are assigned ports in the ephemeral range defined for that cloverleaf server.

In AIX I’ve been able to use lsof to figure out which process has the port locked and tracing it back to the offending interface is how I came to understand the negative impact of using ports in the ephemeral range.

The ephemeral range on a given platform is defined by some configuration method and can be different for a give platform.

First determine how to display what your ephemeral range is defined on the cloverleaf server platform you are using.

For us it is defined from 32K-65K so those ports are off limts and there will be problems if we were to use ports up to 46K.

If I had to use ports up to 46K, then I would need to redefine my ephemeral port range to be something like 46K-65K, which might be the approach to best fit your conflict.

Also, let Mckesson know that they are creating challenges as they encrouch on the ephemeral port range.

It is possible different platforms have different ephemeral port ranges other than 32K-64K but McKesson needs to be made aware to at least watch out for any overlap into the ephermal range.

Another sneaky way a port can be locked that comes to mind is for a rogue interface like from a foriegn systems:

– disaster recovery box

– from a differnet cloverleaf enviroment like DEV/TEST/PROD/

– HA cluster node meant to be shutdown

I have been able to identify the offending foriegn system for this problem using a tcpip trace to see the IP of the foreign system grabbing the port.

One way to help reduce these problems is to make all ports unique across all cloverleaf servers in-house, which is one of our best practices.

Then there is another problem that causes your problem to happen that is worth mentioning.

When an abrupt termination of an interface occurs without a gracefull shutdown of the interface, the listener often thinks it is still connected and up.

This requires the confused interface listener be recycled and then all returns to normal.

An unstable network that has a brief random outages can contribute to this even if it just lasts for a few seconds.

An ubrupt shutdown of a interface during system downtimes or bad interface design lacking graceful shutdown are other causes.

ok, thanks for the time and input. I will do some searching.

You might be running into conflicts with “ephemeral ports”. These are the ports that are fair game for the operating system. This wikipedia article mentions that many Linux’s use the range from 32k to 64k. Your old operating system might have used a different ephemeral port range.

http://en.wikipedia.org/wiki/Ephemeral_ports

It is possible that, while your Cloverleaf connection is down, the operating system comes along and grabs that port from the ephemeral range and assigns it to a “temporary” connection.

I thought there was a way to reserve ports for application use. This discussion thread says both that there is a way and there isn’t.

http://unix.stackexchange.com/questions/15511/how-do-i-reserve-ports-for-my-application

This one mentions putting a reference to the ports in the /etc/services file. That is what I remember. But I don’t really know for sure.

http://www.penguintutor.com/linux/network-services-ports

To see what your ephemeral port range is set to on Linux, you can use this command:

sysctl net.ipv4.ip_local_port_range

You shouldn’t use any ports in that range on Cloverleaf threads that are configured as tcp/ip servers.

Here is how I see the defined ephemeral port range on AIX

which produces this output in my case

Thanks guys, looks like ours is 32K – 61K. i will start changing some ports to see if that helps.

I’m going to pin this conversation. Very helpful info. I’m sure many others will have the same questions.

Greetings,

Another point with the Ephemeral port range is that Cloverleaf uses it for assigning ports on multi-connect server connections.  For us we have developed a script that we run on our test/production servers which creates a csv file that we merge and sort on port to find available ranges.

We have AIX as well and do not assign ports below 8000 or above 32K.

Also:  watch out for Cloverleaf specific app ports like the hostserver uses.

So far in the 13000+ range (think 6.1 uses 13021 now).

If anyone would like the script (ksh guy for AIX but logic could be adapted in another language I’m sure) send me an email.

Enjoy.

We also have a spreadsheet of ports and there were entries made that long before i got here. And i asked our Mckesson support the below:

“I looking at our port listing (which has been around longer than me) but there are entries that say ports 11000-12000 and 13000-13500 are for cloverleaf functions and not to use. Do you know if this is true, at least now?”

And there reply was

“The only restrictions from 6.0 and on are not to use PORTS larger than 65386 or some odd number after 65000.  As far as not using the ones he said no, they are not restricted.”

And i do take everything they tell me with a huge grain of salt.

A whole box.

Greetings,

Check me here other Clovertech folks but there are no ports above 65K for the OS?

Ok..  for the Integrator the only hard ports used to my knowledge are 13000+ and have been assigned incrementally as the releases appear.

These are documented in their Release Notes for each version.

For special firewall settings like the tunnel port in the server.ini Infor support had recommended to us to use 14019 (setting allows display from the testing tools in the IDE to arrive at your location when working remotely).

I have attached our ksh script for your analysis – beware that the hostname in it is ours – a kludge fix as we recently migrated to an HA configuration.

Hope it proves useful to help manage port allocations.

Enjoy.

One and All,

Looks like my attachment had an illegal extension (sh)?

Renamed to .doc and attached.

It is a KSH script.  Rename as you wish.

My apologies.

Enjoy.

McKesson support is wrong.

If you use ports from your ephemeral range, you’re begging for trouble since the OS can dynamically assign those ports whenever it pleases.  If you’re currently using one, I think it will respect that, but as soon as you restart your interface, it’s fair game.

Even outside the ephemeral range, and above the traditional OS-reserved range (up to 1024 in Linux as I recall), there may well be ports that are used by other applications.  I always check if a port is in use before assigning it.

We have a structure for port assignments of little-used port ranges (based on empirical evidence from my Linux systems)

In LIVE:

Inbound: 17000-17999

Localhost: 18000-18999

Outbound: 19000-19999

In TEST:

Inbound: 14000-14999

Localhost: 15000-15999

Outbound: 16000-16999

The code we use for port assignments is below – provided without warranties/guarantees/assurances of any kind – all that standard disclaimer lingo…

It’s built for Linux and relatively un-polished, but has been working well for us.

To use it, you’ll have to edit to add correct path to your NetConfig file(s) and insert your Cloverleaf IP addresses.

Now that is has been mentioned there are no ports that exist above 64K for IPV4, I would like to make a couple of extra comments about that.

Yes I deduced thru observation there are NOT any ports above 65K so I concur.

Before I new this and assigned a port above 65K it seemed to work at first glance until I eventually had a port collision and had to investigate.

What I think happened is that the OS kept subtracting 32K from my assigned port number I assigned above 65K until it got below my ephemeral range of 32K, and then it stopped and used whatever port it landed on which eventually caused a collision.

The amount subratced might of been something other than 32K but it did always keep reducing the port number until it was below my ephermarl port range of 32k-65K.

Russ:

Thanks for your explanation and confirmation on port limits.

Just another point here:  the engine itself uses the ephemeral range

to assign its hcicmd port for a process too.

A very dangerous place to allocate a fixed port assignment in a thread.

Enjoy.

Thanks for posting a script that represent signifcant time and evolution.

I was curious about your approach of using lsof to determine what prots are being actively used that aren’t necessarily defined in a Netconfig or /etc/services file

I decided to run your commands above and compare the sorted -nu portList output from lsof to the method I use to accomplish the same thing via netstat seen below


# create file of netstat ports being used

netstat -a -n [code]
# create file of netstat ports being used

netstat -a -n

Russ –

Part of the difference is the UDP ports, I’m sure.  According to the All Powerful Google,  UDP and TCP ports don’t conflict, so it would seem we can ignore UDP for assignment of TCP ports.

However, I never noticed before, but lsof – on my Linux sysems at least – doesn’t list system ports unless I run it as root.  I guess that makes sense from a security/permissions viewpoint.  For my port assigning purposes, it’s not a problem, but something that I may want to address.

I ran the lsof commands as the hci user.

I got these steps from Phillip at New Hanover. I haven’t tried them myself, but they might be useful for someone someday.

How to Cleanup Ports

 1. netstat -Aan |grep

   * tcp4       0      0  *.             *.*                LISTEN

 2. rmsock tcpcb

   * The socket 0x11e6808 is being held by proccess (hciengine)

 3. set -o vi

 4. ps -ef | grep

   * root 10617004  7799192   0 09:00:32  pts/1  0:00 grep 9634186

   * hci  9634186  7995414   0   Oct 12      – 26:59 /hci/cis5.8/integrator/bin/hciengine -S ambutest -p ambutest_2

 5. kill 9634186

 6. ps -ef |grep 9634186

   * If the process still has not stopped you will see the following

   *  root  8192328  7799192   0 09:01:07  pts/1  0:00 grep 9634186

   *  hci  9634186  7995414   0   Oct 12      – 26:59 /hci/cis5.8/integrator/bin/hciengine -S ambutest -p ambutest_2

 7. kill -9 9634186

 8. ps -ef |grep 9634186

   * root  7471588  7799192   0 09:01:22  pts/1  0:00 grep 9634186

 9. netstat -Aan |grep 2419

In Cloverleaf 6.1, an option has been added for ephemeral ports.

Specifying a Port Range

The Site Init dialog box has a check box where you can configure the ephemeral port range.

Selecting Enable Ephemeral Port Range enables the Port From and Port To fields. The port range must have at least four ports. For example, specifying a range of 10000-10003 allocates ports 10000, 10001, 10002, and 10003.

Valid Port Range From ports are 1025-65532.

Valid Port Range To ports are 1028-65535.

The formula for determining the port range is:

Minimum port number = (total number of processes)X2 + (total number of protocol threads) + 1.