Clovertech

Unable to bind tcp/ip socket: Address already in use

› Clovertech Forums › Cloverleaf › Unable to bind tcp/ip socket: Address already in use

  • Creator
    Topic
  • August 25, 2023 at 10:07 am #120771
    Jerry Sawa
    Participant

      We’re on AIX Cloverleaf 19.1.2.  Occasionally threads get the error “Unable to bind tcp/ip socket: Address already in use“.

      I have “Last Receive” alerts set up that work for our high volume inbound threads, but can’t really rely on Last Receive alert for our low volume threads.

      Is there an alert that can be set up notify us when “Unable to bind tcp/ip socket: Address already in use” occurs?

      • This topic was modified 1 year, 3 months ago by Jerry Sawa.
    • Creator
      Topic
    Viewing 10 reply threads
    • Author
      Replies
      • August 25, 2023 at 10:16 am #120773
        Jeff Dinsmore
        Participant

          Sounds like it may be a port in the ephemeral range.

          What port number is the problem?

          What OS is your Cloverleaf server running?

          Jeff Dinsmore
          Chesapeake Regional Healthcare

        • August 25, 2023 at 10:21 am #120774
          Jerry Sawa
          Participant

            Port number — 53021

            OS is Redhat 8

            It doesn’t happen often, and it is not just one thread that has the issue.  It doesn’t happen very often, maybe once every couple of months.

          • August 25, 2023 at 10:28 am #120775
            James Cobane
            Participant

              You could configure an alert type of “protocol status” == error.  If the thread is unable to bind on the port, the protocol status would go to ‘error’.

              But regarding the port being in the ephemeral range, that could be why you see it happen occasionally.  The ephemeral range is used by the OS to arbitrarily assign ports when needed by the OS.  If it goes to use that port (53021), that could be why you see it happen once in a while.

              We generally only use ports > 10000 and < 32000 for server threads.

              Jim Cobane – Henry Ford Health

              • This reply was modified 1 year, 3 months ago by James Cobane.
            • August 25, 2023 at 10:33 am #120777
              Jerry Sawa
              Participant

                Unfortunately, the Protocol Status is “opening“.   Sorry, I should have mentioned that before.

                How can I find out what the ephemeral range for ports is?

              • August 25, 2023 at 11:20 am #120782
                James Cobane
                Participant

                  I believe on AIX, you can use the command:

                  /usr/sbin/no -a | fgrep ephemeral

                   

                • August 25, 2023 at 11:26 am #120783
                  Jerry Sawa
                  Participant

                    Thanks so much for your help James.  The command didn’t work for me (-bash: /usr/sbin/no: No such file or directory), but I’ll see what I can find.  The range you gave earlier was a big help.

                  • August 25, 2023 at 11:29 am #120784
                    James Cobane
                    Participant

                      We are AIX, and I ran the command to see what it produced:

                      /usr/sbin/no -a | fgrep ephemeral

                      tcp_ephemeral_high = 65535
                      tcp_ephemeral_low = 32768
                      udp_ephemeral_high = 65535
                      udp_ephemeral_low = 32768

                      Hopefully, you can find the command for your environment.

                    • August 25, 2023 at 11:36 am #120785
                      Paul Bishop
                      Participant

                        We have run into this also and it is because of the ephemeral ports (on AIX and same ranges as James’).

                        For AIX you can find what is using that port by using a couple of commands.

                        This returns the PID that is using the port:

                        lsof -i:<port>

                        This will then show what the process is:

                        ps -ef|grep <pid>

                        You can then top the process using the port, start the thread that needs the port, and then restart the process that you just stopped.  Again, this is AIX – not sure what Radhat would use.

                        Paul Bishop
                        Carle Foundation Hospital
                        Urbana, IL

                      • August 25, 2023 at 2:38 pm #120787
                        Jeff Dinsmore
                        Participant

                          If you don’t change the port to something not in your ephemeral range, this will happen again – and likely at the least opportune time possible.

                          In our RHEL system, this command shows the ephemeral port range.
                          > sysctl net.ipv4.ip_local_port_range
                          net.ipv4.ip_local_port_range = 32768    60999

                          The recommendation is to never use ports in this range since, if they’re not in use, the OS can grab them – as you’ve already experienced.  So, you’re playing roulette anytime you bounce a listener using an ephemeral port number.

                          Jeff Dinsmore
                          Chesapeake Regional Healthcare

                        • August 25, 2023 at 2:46 pm #120788
                          Jerry Sawa
                          Participant

                            Thanks everyone.  I never heard anything about ephemeral ports.  I appreciate the input and we will take that into consideration in the future for new threads.  However, we’ve been on Cloverleaf for many years and have many many many many threads.  We’ll see what ports we can change for existing interfaces, but until we do, does anyone know of a way to send an alert when this happens?   Protocol status is “opening”, so we can’t check for “error”.

                            • This reply was modified 1 year, 3 months ago by Jerry Sawa.
                          • August 26, 2023 at 9:30 am #120793
                            Paul Stein
                            Participant

                              If the  thread is typically in an ‘up’ status when this issue doesn’t occur; then you could use a thread protocol status alert to fire  when status does not equal ‘up’

                              • This reply was modified 1 year, 3 months ago by Paul Stein.
                          • Author
                            Replies
                          Viewing 10 reply threads
                          • You must be logged in to reply to this topic.