Reply To: How many messages can the DB/Engine take?

[Dee Davis]

I actually performed a data conversion on the engine with more records than that. I spent quite a bit of time optimizing so we did not overload the database.  There were different stages of the conversion in different processes. Most were fileset-local connections; some were looking at one huge file where records had to be peeled off; others picked up individual files that had been sent out by another stage.

In optimizing inbound fileset-local connections, I looked at: the read interval, max messages and scan interval in the connection set up as well as translation process throttling.  The goal was to have as few messages as possible in the database by the next read but not to leave time where no processing was happening. Also, so that if we had to shut down a process, there was actually some read time for the command thread. Each inbound thread (in different processes) ended up having different “optimal” settings depending on what was happening in the process (parsing the initial extract file, detailed translation, script only, etc).  Examples (read interval, max msg, 3600 scan interval) were 180, 76, 3600; 75, 1000, 3600; 400, 200, 7200.

Bottom line — it can be done without database issues. You just have to work at it.