Cross-Process Routing

[Greg Tataryn]

I know when I first started with Cloveleaf it was generally frowned upon to set up a route from a thread on one process to a thread on another process, though occasionally I believe it was done.

In a recent conversation, it sounded like that is less of an issue these days. As far as best practice goes, I’m looking for opinions and suggestions on this topic. Is it ok to route from one process to another occasionally or is it still better to use intersite routing or use two additional threads to loop the data out and back into the engine to get from one process to another?

Topic

[Robert Milfajt]

In my experience, sometimes it just needs to be done.  Keep in mind this is an AIX slanted opinion, and may not apply to Windows.  It’s a fact that each Cloverleaf process runs on it’s own AIX process.  I’ve found that within a Cloverleaf process there is a limit to the amount of processing it can handle.  Processing can be sheer mesage volume, intensive TCL/Xlate processing or most likely some combination of the two.  I don’t think there is a scientific way to define this limit, and it’s more like a feel type of tuning exercise.  So when a Clovelreaf process hits it’s limit, it starts queuing up messages and delaying processing, this is a great case to route some of the source data to another process so it can be further routed to destinations threads. What this does is give you another AIX process to take some of this load, at the exepense of increasing the work your engine has to do.  In 20 years of Clovelreaf work, I’ve had to do this many times, mostly with ADT as a single source can sometimes go to as many as 90 destinations.

Hope this helps,

Robert Milfajt
Northwestern Medicine
Chicago, IL

[Greg Tataryn]

Thank you for that, Robert. I have noticed the same. In my past organizations we have made use of loop back threads to loop data from the engine back to itself often on the same site just to a different process to help with the load. Intersite threads seem promising but based on information I have received they are still not recommended for high volume (such as ADT) situations.

[Max Drown (Infor)]

It is still  a best practice to keep source threads and destination threads int he same process whenever possible. It’s more efficient processing. This manifests itself as volume increases.

You can’t always avoid it, which is fine. But, as a general rule, keep them in the same process.

-- Max Drown (Infor)

[Charlie Bursell]

To expand on what Max is saying:

When the source and destination are in the same process they share the same memory area.  Thus you pass a message simply by passing a four byte memory location.  When not in the same process the entire message is passed.

It is easy to see that passing 4 bytes vs hundreds or maybe even thousands of bytes is much more efficient especially under heavy loads.

[Peter Heggie]

That has got to be one of the most important performance considerations I’ve ever seen here. Makes me want to reconfigure our threads to use the same process, especially the main HIS-inbound ADT feed.

The inbound thread is already performing all of the translations and pre-translate filtering, so unless there is some complex logic in some outbound thread outbound UPOC, how much could it hurt to put all the outbound threads into the same process as the inbound thread?

Peter Heggie

[Max Drown (Infor)]

Here is a diagram showing the best practice configuration for high volume sites, like ADT sites.

-- Max Drown (Infor)

[Peter Heggie]

Thanks Max – I had seen your diagram before, but now it makes more sense to me.

If there is no smat file/db configured for the input thread of one of those processes, how do you resend messages to a specific destination?

Peter Heggie

[Mark Thompson]

Peter:  Since process 01 raw routes to each of the xlate processes, you can get the message from the fr_his thread and save it to a file.  Then resend the file from the correct process and select the desired destination.

Max: I have an old “best practice” note that says to limit 1 inbound thread to 8 outbound threads.  I believe that is because (at that time) the messages to all destinations fail if a single route fails.  Is there still a reason to limit the number of outbound threads from a single input?

- Mark Thompson
HealthPartners

[Max Drown (Infor)]

Since the receiving process sends the messages to the xlate processes in a raw format, unchanged, you get the messages from that SMAT file and then resend it to the correct xlate process. The resend dialog has a field that lets you restrict the destination to one or more specific threads.

If you find yourself doing a lot of resending to one of the destinations, it’s OK to turn on SMAT for that process or some of the threads. You generally want to reduce the I/O as much as you can, but sometimes it can be worth it to turn on SMAT.

If the xlate process is in another site, then I would turn on SMAT.

You can also use Global Monitor for searching SMAT and resending messages.

There is no best practice for the number of threads to include in each xlate process because there are so many factors involved.

1. The priority of the destination application

2. The volume of messages bother daily and during peak loads

3. The number of CPU’s

4. What each interface is doing, heavy tcl, db work, web service calls, etc.

5. Combing multiple interfaces into one destination tcp port

6. Visible complexity of the NetMonitor when supporting the interfaces

7. Etc.

Design each xlate process with these things in mind and then move threads around as you see fit. For example, its a good idea to isolate a trouble connection, a high priority connection, connections that have trouble reestablishing connectivity when bounced, or connections with a lot of code changes.

-- Max Drown (Infor)

[Peter Heggie]

thanks – I need to think this through.

So if I want to resend a message to destination thread ‘to_out04’ in process02, and its sending thread ‘fr_int01’ does not have smat, then I have to go back to process01 and resend from ‘fr_his’ ? How can I specify the destination thread ‘to_out04’ from the dialog for thread ‘fr_his’ in the other process?

Sometimes we resend messages in the outbound thread, but usually we resend from the inbound thread so we can go through the translate again.

Peter Heggie

[Max Drown (Infor)]

From SMAT, you would search for the message you want to resend and then you would choose any thread in that site to resend the message to. This thread can be inbound or outbound. If it is an inbound thread and you only want to resend the message through the xlates to one (or more) destination, then you can specific those destination in the resend dialog. The resend dialog also let’s you specify which state to use for resending (Pre/Post In/Out).

-- Max Drown (Infor)

[Peter Heggie]

The smat file specified in the outbound thread would hold the version of the message that existed after translation. If the application told us that we need to resend messages because, for example,  some lookup table value had changed in the application, and we needed to update our table and run the messages through the translate again, the version of the message in the outbound thread smat file would not be appropriate. We would need to send the message as it first came into the inbound thread.

Peter Heggie

[Max Drown (Infor)]

Resending pre-translation is extremely common.

Process 01 will have inbound SMAT turned on. Process 01 sends messages RAW and unchanged to the xlate processes. If you were to turn on inbound SMAT in the xlate processes, you’d be logging the exact same message twice.

You can grab the inbound message from Process 01 and then resend it to any thread in the site.

-- Max Drown (Infor)

[Peter Heggie]

sorry to belabor the point – if I resend from smat in process01, how can I direct that to ‘to_out04’?

Peter Heggie

[Max Drown (Infor)]

You just choose the thread from the resend dialog. There’s a drop-down list containing all of the thread names in the site.

You can also resend to a file, move the file around, edit it, etc. and resend the message using the resend dialog in the NetMonitor. Or you can use a script to resend the message.

-- Max Drown (Infor)

[Peter Heggie]

ok so you are saying that because the messages are raw routed, we can pull up the smat file for ‘fr_his’ , but select as the sending thread ‘fr_int01’ and then that allows me to select ‘to_out04’ in the Destination field.

Peter Heggie

[Max Drown (Infor)]

Peter Heggie wrote:

ok so you are saying that because the messages are raw routed, we can pull up the smat file for ‘fr_his’ , but select as the sending thread ‘fr_int01’ and then that allows me to select ‘to_out04’ in the Destination field.

That is exactly what I am saying. 🙂

-- Max Drown (Infor)

[Peter Heggie]

thank you!

Peter Heggie

[John Douet]

Greg – we have seven sites with a bit over 72 processes and who knows how many threads, and use intersite and interprocess threads wherever it makes sense to do so. We route just about every type of message we get this way, usually to help maintain a logical flow and thread grouping, including ADT.

We converted from AIX to Windows about 18 months ago, so I can’t tell you if the practice has any positive impact on the Windows environment, we don’t experience any issues that I can speak of that can be attributed to this type of routing.  We’re currently running 6.1.1, are a 530-bed hospital, and average roughly 13,000 messages daily through our main ADT/orders feed, though total traffic is much higher due to all of the ancillary systems.

[Ben Ware]

Max, we are using a very similar setup to the one in your diagram.

[Max Drown (Infor)]

Thank you for logging the issue. Let’s see what they say. Please keep us updated here.

-- Max Drown (Infor)

[Ben Ware]

Unfortunately Infor is refusing to look at the issue any more unless I reintroduce the issue into our production environment with logging verbosity turned all the way up.

[Ben Ware]

I believe I have isolated what triggers the issue.

[Max Drown (Infor)]

Ben, this is very helpful troubleshooting. Would you please submit this information to your Infor Xtreme ticket?

-- Max Drown (Infor)

[Max Drown (Infor)]

Peter Heggie wrote:

sorry to belabor the point – if I resend from smat in process01, how can I direct that to ‘to_out04’?

There are a number of ways to do this depending on the use case.

Typically you would not resend from Process01, but instead for inbound SMAT in the xlate processes.

If for some reason that does not work, you may want to save the messages from a file in Process01 and resend to ProcessN using that file.

-- Max Drown (Infor)

[Author]

Replies