Reply To: tcl help to count no of messages and append to the top

[Kevin Kinnell]

Ray

I’m still not quite clear on the sequence here, I think.  I was presuming that you are counting the messages that are in the separate files which you ftp out, and generating a new log file which is then ftp’d out as well.  What I think now is that there is a log file on the ftp server which will be picked up once an hour, but which needs to be kept updated continuously.

Does the log file continuously grow, with insufficient space for it locally?   Log files can’t grow forever–how often does the log get ‘zeroed’ and a new log started?   How reliable is the retrieval time?  Does it happen at the same time every hour?

I’d think retrieving/processing a file via ftp in a tps would be too darned slow.  I think what I’d do if there were no way around it would be a slight modification of what Michael suggested: generate your log file locally, and a separate file with the current count (the hci extensions have a set of Ctr* procs for manipulating counter files.)   First you’d append the new messages to the message file, updating the count.  Open your new log, write the count, then close it and cat the messages onto it  and ftp the result to the server.

Now, you could ftp the log file back to your site from the server.  You’d slow things down and introduce a bigger window in the implicit race condition–what if the log gets picked up while you’ve got it?  But, you’d still have to be doing all the processing locally.  You actually just narrow the window on the race condition a bit by keeping the data local if there is a copy on the server.  So Michael’s unmodified solution is actually your best bet: generate everything locally, and don’t hand it to the server until it’s needed.  This completely avoids the ‘two copies of the “same” data’ problem, which is an huge problem that you want to avoid like the plague.

If you can’t control when the file will be picked up, you can use clock to prevent your proc from ftp’ing until you are ready (and avoid the Windoze scheduler altogether.)  Which may, now that I consider it, be what you wanted to know in the first place.  You would still have to make sure that the previous file has been retrieved before you sent the new one (assuming that retrieval removes or zeroes the log file.  Does it?)

Whatever you do, for this kind of thing, you really need to build in a bunch of tests for not re-transmitting information that’s already been retrieved (I speak from bitter experience.)  Never, ever have more than one copy of the data available for different entities to process.  There is no way to guarantee things will be in sych if you do.