unable to realloc *** bytes, where are the limitation

Homepage Clovertech Forums Read Only Archives Cloverleaf Cloverleaf unable to realloc *** bytes, where are the limitation

  • Creator
  • #49619

    Cloverleaf 5.4.1 on Win2003

    I get the msg unable to realloc 248619850 bytes

    and the process goes down.  Can anyone explain where the limitations are. There should be enough memory om the server. It happends when I parse big (86Mb) files in Tcl. It seems like the file takes up 4 times the size of the file in memory.

    Any help are appreciated


Viewing 4 reply threads
  • Author
    • #62746
      David Barr

      We just looked at a very similar problem recently.  Our Cloverleaf processes were dying with the same error message when we tried to pass very large messages (over 60MB).  By changing the coding of our Tcl procs, we were able to avoid this problem.  We made sure that for very large messages, we didn’t do any “msgget” calls to load extra copies of the message into memory.

      Our problem happened on AIX, not Windows.  We were unable to determine where the memory limit was coming from.  My guess is that it is due to the compiler settings that Quovadx used when they compiled TCL.  The memory limit for us seemed to be about 500MB.  After our coding changes, we were able to pass messages up to about 150MB through the engine.

      I’d recommend changing how your code works so that you don’t hold all of the message in memory.  Sometimes it’s possible to work with your data a line at a time or a megabyte at a time.

    • #62747
      Steve Carter


      I believe the issue in the in the UNIX environment has to do with the ulimit settings for the hci user.  Run ‘ulimit -a’ from the command line and you should see something like this:

      $ ulimit -a

      time(seconds)        unlimited

      file(blocks)         2097151

      data(kbytes)         1572864

      stack(kbytes)        262144

      memory(kbytes)       262144

      coredump(blocks)     2097151

      nofiles(descriptors) 10000

      I think it’s the ‘memory’ parameter that limits how much memory the hci user is allocated. The ‘file’ parameter limits the size of a file that the hci user can create.

      By changing your code (i.e., reading files line by line, writing files line by line, etc.), you reduce the memory usage, therefore, solving your problem.


    • #62748
      David Barr

      Steve Carter wrote:


      I believe the issue in the in the UNIX environment has to do with the ulimit settings for the hci user.  Run ‘ulimit -a’ from the command line and you should see something like this:

      You’re right, that can be a limitation of process memory usage, but try this experiment.  In one of your tps procs, write “exec ulimit -a >&@ stdout;”.  If you look at the output of ulimit in your process log, you’ll see that Cloverleaf automatically sets the data and memory limits to “unlimited” after the process is started.

    • #62749
      Michael Hertel

      We had this issue when the smat file became too big.

      Are you saving messages on the thread?

    • #62750

      I cut this down to 1 ib thread and 1 ob thread, one tcl in the inbound tps that reads the entire file into a dom, this means it reads it into memory.

      No recovery db, no SMAT, fileset local in both threads.

      When this process is running just doing nothing it’s hciengine.exe proc uses about 120mb mem. When I put a small file through the thread the memory use rise and falls back to 120 mb again. When I put a medium file throug the mem usage rises more and falls back to 120mb again. When I try to put a huge file through the memory usage rises, up to some limit and the process goes down and writes to the log ‘unable to realloc xxxxx bytes’. It seem like when hciengine.exe tries to use more than 500 Mb memory it goes down.

      On this test server there are about 1.5 Gb free memory (3Gb total) on the produktion server, where the same thing happens there are 8Gb total. I’ve talk to some hard core WinServer guys and they tell me that the only limitation in windows is 2Gb because it’s a 32 bit system, and no limitation on user as in Unix.

      So can anyone show me a screendump af a task manager with a hciengine.exe that uses more than 500Mb??

Viewing 4 reply threads
  • The forum ‘Cloverleaf’ is closed to new topics and replies.

Forum Statistics

Registered Users
Topic Tags