Unable to bind tcp/ip socket: Address already in use

[Jerry Sawa]

We’re on AIX Cloverleaf 19.1.2.  Occasionally threads get the error “Unable to bind tcp/ip socket: Address already in use“.

I have “Last Receive” alerts set up that work for our high volume inbound threads, but can’t really rely on Last Receive alert for our low volume threads.

Is there an alert that can be set up notify us when “Unable to bind tcp/ip socket: Address already in use” occurs?

• This topic was modified 1 year ago by Jerry Sawa.

Topic

[Jeff Dinsmore]

Sounds like it may be a port in the ephemeral range.

What port number is the problem?

What OS is your Cloverleaf server running?

Jeff Dinsmore
Chesapeake Regional Healthcare

[Jerry Sawa]

Port number — 53021

OS is Redhat 8

It doesn’t happen often, and it is not just one thread that has the issue.  It doesn’t happen very often, maybe once every couple of months.

[James Cobane]

You could configure an alert type of “protocol status” == error.  If the thread is unable to bind on the port, the protocol status would go to ‘error’.

But regarding the port being in the ephemeral range, that could be why you see it happen occasionally.  The ephemeral range is used by the OS to arbitrarily assign ports when needed by the OS.  If it goes to use that port (53021), that could be why you see it happen once in a while.

We generally only use ports > 10000 and < 32000 for server threads.

Jim Cobane – Henry Ford Health

• This reply was modified 1 year ago by James Cobane.

[Jerry Sawa]

Unfortunately, the Protocol Status is “opening“.   Sorry, I should have mentioned that before.

How can I find out what the ephemeral range for ports is?

[James Cobane]

I believe on AIX, you can use the command:

/usr/sbin/no -a | fgrep ephemeral

 

[Jerry Sawa]

Thanks so much for your help James.  The command didn’t work for me (-bash: /usr/sbin/no: No such file or directory), but I’ll see what I can find.  The range you gave earlier was a big help.

[James Cobane]

We are AIX, and I ran the command to see what it produced:

/usr/sbin/no -a | fgrep ephemeral

tcp_ephemeral_high = 65535
tcp_ephemeral_low = 32768
udp_ephemeral_high = 65535
udp_ephemeral_low = 32768

Hopefully, you can find the command for your environment.

[Paul Bishop]

We have run into this also and it is because of the ephemeral ports (on AIX and same ranges as James’).

For AIX you can find what is using that port by using a couple of commands.

This returns the PID that is using the port:

lsof -i:<port>

This will then show what the process is:

ps -ef|grep <pid>

You can then top the process using the port, start the thread that needs the port, and then restart the process that you just stopped.  Again, this is AIX – not sure what Radhat would use.

Paul Bishop
Carle Foundation Hospital
Urbana, IL

[Jeff Dinsmore]

If you don’t change the port to something not in your ephemeral range, this will happen again – and likely at the least opportune time possible.

In our RHEL system, this command shows the ephemeral port range.

> sysctl net.ipv4.ip_local_port_range

net.ipv4.ip_local_port_range = 32768    60999

The recommendation is to never use ports in this range since, if they’re not in use, the OS can grab them – as you’ve already experienced.  So, you’re playing roulette anytime you bounce a listener using an ephemeral port number.

Jeff Dinsmore
Chesapeake Regional Healthcare

[Jerry Sawa]

Thanks everyone.  I never heard anything about ephemeral ports.  I appreciate the input and we will take that into consideration in the future for new threads.  However, we’ve been on Cloverleaf for many years and have many many many many threads.  We’ll see what ports we can change for existing interfaces, but until we do, does anyone know of a way to send an alert when this happens?   Protocol status is “opening”, so we can’t check for “error”.

• This reply was modified 1 year ago by Jerry Sawa.

[Paul Stein]

If the  thread is typically in an ‘up’ status when this issue doesn’t occur; then you could use a thread protocol status alert to fire  when status does not equal ‘up’

• This reply was modified 1 year ago by Paul Stein.

[Author]

Replies