translation performance tuning

[John B Frame]

We have a data file with 30,000 frl messages (one message per row).

The inbound thread reads this file, does a single translation for each message, and sends the result (which is an xml message) to an outbound thread which writes each message as a file to a directory on the local file system.

The speed is less than 2,000 messages per hour. It should be many times faster.

I would love to hear about performance best practices, hints, and tricks.

Thanks

John Frame

jbframe@lasd.org

Topic

[Robert Kersemakers]

Hi John,

Take a look at the ‘Scheduling’ options at the FileSet/FTP Local properties.

You should increase the ‘Max messages’ from 1 to about 100 (just a wild guess on my part). This means every 5 (= Read Interval) seconds 100 messages will be processed, instead of just one.

Zuyderland Medisch Centrum; Heerlen/Sittard; The Netherlands

[Bill Bertera]

are you reading the whole file at once, or 1 line at a time?

I would pick up the whole file, and either:

1. split the file into a many messages, 1 for each line, then Xlate each.

or

2. send the whole file to the Xlate, then use CONTINUE to spit out a new message for each line.

I think option 1 will be faster.

[Rob Abbott]

Also – if your XML description is using a schema vs a DTD, I recommend that you use a DTD if at all possible.  The XML parser in 5.3 and below is significantly slower if a schema is used.

This issue will be addressed in the 5.4 release.

Rob Abbott
Cloverleaf Emeritus

[Author]

Replies