Fileset local fills up recovery DB

[Mike Strout]

I ran into a situation where I needed to resend two months of charge messages (about 20k per day) through Cloverleaf into three databases. Because we are on 6.1.2, doing so directly from SMAT DB files is painfully slow, I thought the best way to accomplish the resend would be to resend each day of messages from SMAT to a file and then create a fileset_local thread to feed them over to the databases.

All seemed to be going as planned with the inbound thread chewing through about 40 messages per second. The GUI showed no backup going to the downstream threads, though the routes did stay pretty backed up.

I made it through about 600,000 messages before the inbound process died because the RDB was full. I don’t understand why these wouldn’t be dumped down to the DBs at the very least at the end of each new file, but ideally at the end of each bite of the file (5 read interval, 200 max messages, 30 second scan interval).

What did I do wrong?

Topic

[James Cobane]

It sounds like things were queuing up in post-xlate, and never really getting pushed to the outbound thread, thus they are in the recovery database at state 7.  You may need to implement some “Translation Throttling” or scale back your read/scan times to let the engine process what it has consumed.  The engine will give priority to consuming the messages.  Sometimes it’s trial and error on the read/max/scan settings to get a good balance.

Jim Cobane

Henry Ford Health

[Mike Strout]

Thanks James. I thought 40 messages per second looked too good to be true. I didn’t realize Cloverleaf would behave so async’ly. I guess I need to throttle back a bit.

[Author]

Replies