Large XML messages crashing process

[Tiffany Bohall]

For the last couple of weeks, we have been receiving XML messages (via TCP/IP) that can be in excess of 100MB due to backloading data.  These are GL transactions that get parsed into a VRL outbound.  We can receive anywhere from 20-100 XML messages all varying in size but when we receive a message over 100MB, our process crashes.  This message has to be removed from the RDB before the process will successfully come back online and continues receiving.  Our logs are showing core dumps.  Has anyone else had experience processing large XML messages in the engine?  The vendor is unable to make many modifications and the backloading of data will proceed for an unspecified period of time.  We are on v20.1.1.  Thank you for your input.

Topic

[Charlie Bursell]

What OS?

[Tiffany Bohall]

We are on AIX v 7.1.0.0.

[Jerry Sawa]

Does the site have a lot of threads?  Is the process dedicated to this particular thread or does it have more?  Just trying to get a feel as to how busy the site and process are.

Not sure if this helps as I’ve never dealt with msgs of that size, but we do huge backloads from our TEST server that has a site/process dedicated just for backloads.   Not a lot going on in our test server so memory not as much of an issue and that way we’re not impacting the Production server.

[Tiffany Bohall]

Thank you for responding.  We have 38 threads on the site and a majority of them are fileset-local, only processing data once/week/month, etc.  The process is dedicated to these two threads.  The problem is that these backloaded transactions are mixed with non-backloaded transactions and the vendor thus far has been unwilling to split them.

Per my open issue with Infor, Gotham has advised us to write the messages to a file and then split them out based on message size.  The receiving side will have more files to pick up but as long as it stops crashing us, it is an acceptable compromise.  We are currently in testing mode.

[Charlie Bursell]

You might want to take a  look here:
https://www.ibm.com/docs/en/aix/7.2?topic=memory-interprocess-communication-limits

Also, on older AIX boxes there were system values that had to be manipulated for very large messages.  I’m not sure that is still an issue.   I can’t remember the values that had to be changed (old man memory) but I am sure Cloverleaf Support could help.

[Tiffany Bohall]

Thank for this information, I will review with our server admins.

[Tiffany Bohall]

This probably will not surprise you, but this is exactly what the issue is.  Thank you again for the article and stopping the spinning so we could focus on another solution to try and resolve this once and for all!  I appreciate your time and feedback!

[Jerry Sawa]

Tiffany – I’m very curious about this, once resolved please let me know what you had to do.

[Tiffany Bohall]

This has been an interesting ride and I am so glad it is coming to an end hopefully in the next few weeks!  Unfortunately, since Cloverleaf AIX is a 32-bit application with smaller memory addressing than Windows and Linux this is the cause of our issues here.  We knew there was a 32-bit  limitation before this issue came up but it was only seen via our inbound SMAT files that consumed so much data we often lose the file in archive and it results in 0kb files.  We did not associate this problem to large inbound messages and had a very difficult time proving so.  Since we will not be upgrading our OS in at least the next year, we have had to take a solution outside of the engine.  We are using a Windows server and wrote a .net solution that will take the messages via SFTP, parse them into a single .csv for SFTP to our downstream app.  It is not idea and there is hesitation going outside of the engine for this, but it is what we have been dealt.

[David Dillard]

Did you try turning on the Disk-Based Queuing for the process?

We had a problem with large PDF messages causing memory issues but they seemed to go away after we turned this on.

The AIX 32-bit limit is a pain

[Author]

Replies