Guideline for using sleep in Tcl in a site

[Bob Richardson]

Greetings,

We are looking to set up an inbound fileset thread for files delivered by an external ftp process to our Cloverleaf engine.  These files are to be converted to XML and parked in an outbound directory by another thread in the same process.  Just these two threads in the process.

To avoid the problem of grabbing these files before the ftp process finishes writing them to the inbound directory, we were toying with an idea to write a tps directory parse tcl proc with a loop that checks file sizes, sleeps a few seconds, checks again until they are equal: first check, sleep n seconds, second check – equal? pass the file list back to engine.

Normally we do not put sleep into our tcl for running engines (processes).  Our old knowledge as to “why” is a bit lost.  So… we are of two minds: a sleep in a single process only affects that process in a site but not any other processes.  They continue to get cpu cycles – we figure.  But some little daemon bugs me that the entire site may be affected too.

Bottom line: does a tcl sleep in one process in a site affect the other processes in this same site?  that is, do they continue to run or halt?

Thanks for your response in our quest to “reload” ancient knowledge!!

Topic

[Charlie Bursell]

The simple answer to your question is no, a sleep in one process will not affect another.

The reason we do not like sleep commands is it will block.  But as you say since nothing else is in the process it won’t matter.

There are two ways I handle your problem.  First, if your vendor would FTP to a temp directory and then when finished move the files to the Cloverleaf directory would be the simplest.  A file move is an atomic operation if done on the same file system.  Only the inode is changed, the file is not moved.

I have a standard Directory Parse proc I use which instead of using sleep, uses the scan time.  It will look at the files proposed by the engine and if its the first time seen, simply size the file and save it in a global.  Then on the next scan, if the sizes are still the same, fetch the file.  The assumption is if the size is the same between 2 scans. the file is stable.

Your idea of sleeps will work also since it the only thing in the process.  Of course even if it is the only thing it will still block ongoing translations, etc.

[David Barr]

A sleep in a Cloverleaf thread will block all threads in the process, but it will not block other processes.  This means that your sleep will block the command thread.  It also means that if you put other threads in the same processes, they won’t get processing time.  Your process will also not be able to respond to process stop and thread stop requests for the duration of the sleep.  This probably isn’t a big deal because it is only for a few seconds, and only when you are waiting for a file transfer to complete.

I don’t really like the idea of putting a sleep in the directory parse proc.  An additional drawback is that your sending application may pause before the file transfer is completed, and you’d think the file was complete.  If you’re going to base your decision based on a lapse in time from the last write to the file, you might as well just check the file modification time and ignore the file if it has been modified in the last few seconds (or whatever your criteria is).

I prefer getting the sender to rename the file after the transfer is complete.  Another option is to have your directory parse proc check to see if the file is open for writing by another process before it returns the filename.  The file will probably be in use by the ftpd process. You can call the “lsof” utility to check on this (assuming you are on Unix).

[Bob Richardson]

Folks,

Thanks for the information – clear and extremely helpful.

Charlie, I would be interested in an example of your proc.

By the way, this would be a great topic for the Level III Cloverleaf class as we did not cover anything like this question.  I understand that one of the goals in this class is ‘engine design’.

Again, thanks for the illumination guys !!

[Charlie Bursell]

Send me an e-mail and I’ll send you a copy.

I don’t know when you attended Level 3 but it is certainly covered now.

FYI for the whole community.  Level 3 is always taught by a very senior Implementation Engineer and not by the trainng folks.  Right now it is usually taught by me or Viken Ohannessian.

In addition to covering advanced Cloverleaf topics, we try to impart the benifit of our experiences.  I would guess that about 0% of what I cover is not on the slides but more in the line of tips, trick, traps, etc.

We have done away with a meaningless written test.  The Level 3 test is now strictly an implementation task similar to that you might get in the real world.

[Richard Hart]

Bob.

I’ve only used this for load testing, where we send production message at the same time as they were sent in production.

This may be useful

I created one proc, used in the the file set (fileset_ibdirparse) and inbound (sms_ib_data).

The inbound data receives real and ‘timer’ messages.

If a ‘timer’ message is received, all queued message are checked to see if we can deliver them and if the list is clear, the timer flag is cleared.

If a real message is received, the delivery time of the message is checked and it is either sent on, or stored in a list and a ‘timer’ flag is set.

The fileset code runs at the prescribed time and, if the ‘timer’ flag is on sends a timer message which kicks the ibnound to check its ‘queued’ messages or send real messages.

[Author]

Replies