- This topic has 10 replies, 4 voices, and was last updated 11 months, 2 weeks ago by
.
-
Topic
-
For the last couple of weeks, we have been receiving XML messages (via TCP/IP) that can be in excess of 100MB due to backloading data. These are GL transactions that get parsed into a VRL outbound. We can receive anywhere from 20-100 XML messages all varying in size but when we receive a message over 100MB, our process crashes. This message has to be removed from the RDB before the process will successfully come back online and continues receiving. Our logs are showing core dumps. Has anyone else had experience processing large XML messages in the engine? The vendor is unable to make many modifications and the backloading of data will proceed for an unspecified period of time. We are on v20.1.1. Thank you for your input.
-
Replies
-
-
-
-
Does the site have a lot of threads? Is the process dedicated to this particular thread or does it have more? Just trying to get a feel as to how busy the site and process are.
Not sure if this helps as I’ve never dealt with msgs of that size, but we do huge backloads from our TEST server that has a site/process dedicated just for backloads. Not a lot going on in our test server so memory not as much of an issue and that way we’re not impacting the Production server.
-
Thank you for responding. We have 38 threads on the site and a majority of them are fileset-local, only processing data once/week/month, etc. The process is dedicated to these two threads. The problem is that these backloaded transactions are mixed with non-backloaded transactions and the vendor thus far has been unwilling to split them.
Per my open issue with Infor, Gotham has advised us to write the messages to a file and then split them out based on message size. The receiving side will have more files to pick up but as long as it stops crashing us, it is an acceptable compromise. We are currently in testing mode.
-
-
You might want to take a look here:
https://www.ibm.com/docs/en/aix/7.2?topic=memory-interprocess-communication-limitsAlso, on older AIX boxes there were system values that had to be manipulated for very large messages. I’m not sure that is still an issue. I can’t remember the values that had to be changed (old man memory) but I am sure Cloverleaf Support could help.
-
-
-
This has been an interesting ride and I am so glad it is coming to an end hopefully in the next few weeks! Unfortunately, since Cloverleaf AIX is a 32-bit application with smaller memory addressing than Windows and Linux this is the cause of our issues here. We knew there was a 32-bit limitation before this issue came up but it was only seen via our inbound SMAT files that consumed so much data we often lose the file in archive and it results in 0kb files. We did not associate this problem to large inbound messages and had a very difficult time proving so. Since we will not be upgrading our OS in at least the next year, we have had to take a solution outside of the engine. We are using a Windows server and wrote a .net solution that will take the messages via SFTP, parse them into a single .csv for SFTP to our downstream app. It is not idea and there is hesitation going outside of the engine for this, but it is what we have been dealt.
-
-
-
- You must be logged in to reply to this topic.
