Error Msg: Unable to obtain a tcp/ip socket: Too many open f

Clovertech Forums Read Only Archives Cloverleaf General Error Msg: Unable to obtain a tcp/ip socket: Too many open f

  • Creator
    Topic
  • #49056
    Darren Powell
    Participant

      Hello All, I have a TCP receive thread throwing the following error:

      Error Msg: Unable to obtain a tcp/ip socket: Too many open files

      Anyone ever deal with this before?

      Thanks!

      Darren Powell

      darren.powell@jhsmh.org

      (502)560-8451

    Viewing 4 reply threads
    • Author
      Replies
      • #60578
        Rick Brown
        Participant

          I believe you are probably on AIX, correct?

          I think this is usually related to your /etc/security/limits settings.

          There is a value called nofiles = 10000.

          I think 10000 is a good starting point but can vary or be increased based on hardware.

          Thanks,

          Rick

        • #60579
          Steve Carter
          Participant

            All too many times……

            In my environment this is generally caused by port conflicts.  Each time a thread attempts to bind to a port, it creates a file handle.  Depending on how often it retries the bind and what your nofiles parameter is set to, you’ll start getting this error.  For example, if you’re thread is set to retry every 5 seconds and your nofiles parameter is set at 2000.  You’ll run out of file handles in just over 2 hours and 45 minutes (10,000 seconds).

            In my case, I had developers assign client ports in the ephemeral range which were causing port conflicts after a reboot.

            When Cloverleaf fails to bind to a port, it does not release the file handle.  It just continues to use handle after handle until it runs out.  In my opinion, this is a bug in Cloverleaf.  The thread should just release the file handle and move on.  Unfortunately, that isn’t the case.

            Hope this helps.

            Steve

          • #60580
            Michael Hertel
            Participant

              Talk to your Unix administrator, there is a way to find out how many open files there are and which pid owns them. You can find out which process or processes are being a hog.

              In our case, in an older version there was a bug and the ftp interfaces would do what Steve described. So I would have to find the offending process, stop it and it’s associated threads, restart and life was wonderful, at least until it happened again.

            • #60581
              Russ Ross
              Participant

                I vaguely recalled having used an AIX command that would list what files are open by what processes.

                After some digging I uncovered the fuser command.

                Here is the first part of the man pages for fuser:

                Code:

                fuser Command

                Purpose

                Identifies processes using a file or file structure.

                Syntax

                fuser [ -c | -d | -f ] [ -k ] [ -u ] [ -x ] [ -V ]File …

                Description

                The fuser command lists the process numbers of local processes that use the
                local or remote files specified by the File parameter. For block special
                devices, the command lists the processes that use any file on that device.

                Each process number is followed by a letter indicating how the process uses the
                file:

                c Uses the file as the current directory.

                e Uses the file as a program’s executable object.

                r Uses the file as the root directory.

                s Uses the file as a shared library (or other loadable object).

                The process numbers are written to standard output in a line with spaces between
                process numbers. A new line character is written to standard error after the
                last output for each file operand. All other output is written to standard
                error.

                The fuser command will not detect processes that have mmap regions where that
                associated file descriptor has since been closed.

                Below is a screen shot of a simple usage of fuser:

                Russ Ross
                RussRoss318@gmail.com

              • #60582
                Michael Hertel
                Participant

                  I was thinking more of lsof. (Maybe “lsof -a”)?

                  You won’t know what the file name is that is open (because it may be a device instead of a file) but if you run it thru “less” or “more” you’ll see certain pids are bigger pigs than others.

                  Someone else may be able to explain this better than I can.

              Viewing 4 reply threads
              • The forum ‘General’ is closed to new topics and replies.