- This topic has 5 replies, 5 voices, and was last updated 16 years, 3 months ago by
.
-
Topic
-
I have unique problem. First I’m running AIX 5.3 cloverleaf 5.5 Rev1.
This is a one time conversion process. I have 10 years of data about 2 million messages to send.
I need to limit the number of message I send to about 60,000 per day.
The outbound system can generate 100,000 plus per day.
If you were given this issue how would you limit the outbound message to what the receiving system can handle ?
Any thoughts suggestion are welcome.
-
Replies
-
-
I had a similar requirement for an outbound thread that the recieving system could not handle the volume of messages the sending system was supplying. I added a ‘sleep’ command to a tclproc in the SEND OK UPOC. A couple caveats – Be careful you are not tying up the process manager if there are other threads in the process. Also be careful you will not have too many messages queueing up in the engine.
Here is the proc…
Code:######################################################################
# Name: tps_msg_throttle
# Programmer: Scott R. Lee
# Purpose: Slow down messages going out to EDM. Charges from Affinity
# can be sent too quickly for EDM to handle them and
# other messages (mainly ADTs) get queued up behind them.
#
# Edits: 6/4/07 – Scott R. Lee
# Changed sleep time from 1 second to .5 seconds.
# 6/6/07 – Scott R. Lee
# Changed sleep time back to 1 second.
#
# UPoC type: tps
# Args: tps keyedlist containing the following keys:
# MODE run mode (”start”, “run” or “time”)
# MSGID message handle
# ARGS user-supplied arguments:
#
#
# Returns: tps disposition list:
#
#proc tps_msg_throttle { args } {
keylget args MODE mode ;# Fetch modeset dispList {} ;# Nothing to return
switch -exact — $mode {
start {
# Perform special init functions
# N.B.: there may or may not be a MSGID key in args
}run {
# ‘run’ mode always has a MSGID; fetch and process it
keylget args MSGID mh
echo sleeping 1
sleep 1
lappend dispList “CONTINUE $mh”
}time {
# Timer-based processing
# N.B.: there may or may not be a MSGID key in args
}
shutdown {
# Doing some clean-up work
}
}return $dispList
}
HTH
Scott
-
Are the messages in a file and Newline delimited? If they are in a file and you are on UNIX, you can use the UNIX split command to create smaller files to resend. I recently did a conversion and found that the receiving system would create files with particular extension for the type of message it was receiving. I scripted individual messages into multiple zip files that we assigned a person to extract every few minutes to allow production messages to continue to flow with minimal delay. This removed the overhead off of the engine. If I had left the processing on the engine, it would have taken days with current production backing up…. This happened to be GE Centricity so it depends on your destination system for similar ways to address.
-
John: Look at my posts at URL:
http://clovertech.infor.com/viewtopic.php?t=1621&http://clovertech.infor.com/viewtopic.php?t=1621&” class=”bbcode_url”> where I describe how we handle downtime mode and resending and throttle massive amounts of messages when the foreign system comes back up.
Russ Ross
RussRoss318@gmail.com -
Hi John. We performed a data load of millions of messages a few years ago. We were sending to an application that internally queued message before processing, so we didn’t get an application ACK until later.
This is how our solution worked .
The messages were in zipped (gzip on Unix) SMAT files that we had cycled daily over the years.
We wrote some TPS that scanned the data directory and returned a list that was either blank or contained one file to read.
A blank list was returned if:
a ‘stop’ flag was set (a simple file existence check); or
the difference between the message sent and ACKs received was greater than a threshold value; or
a SMAT file was the next file and this was unpacked in the background.
A file was returned if:
a normal file was.
This ensured that we had easy control to stop the message flow – once the code was scanning the directory.
We ensured that the processing application queue (MSMQ) was never too large as a crash would lose the messages!
Messages came from our SMAT files, so we could use ‘cron’ or ‘at’ to load more SMAT files in.
We had this going for about 3 months!
-
John: Richard’s post reminded me of a point worth mentioning and you may know this already but not everyone will.
When I end up with one very large new-line terminated file of messages to resend, I can use the AIX/UNIX split command to chop the file up into smaller files and send those one at a time.
For example:
split -10000 bigFile
will split the bigFile into smaller files with 10,000 messages in each of the smaller files.
Here is my generic resend script ( resend_que.ksh ) that I call to automate doing most of the work of resending multiple files:
Code:#!/usr/bin/ksh
# Begin Module Header ==============================================================================
#
#
#——
# Name:
#——
#
# resend_que.ksh
#
#———
# Purpose:
#———
#
# will resend the next que file in name sorted order
#
# the requirements are that the resend process is already running and the outbound thread is UP
#
#——-
# Notes:
#——-
#
# You need to at least manually start the resend process and confirm the resend_thread is up
# before invoking this script
#
# Have the calling script export/set these arguements before invoking this script.
#
# Here is an example:
#
# export que_dir=”$HCISITEDIR/data/golive_archive”
# export load_dir=”$HCISITEDIR/data/load_rxtfc_adt_9027″
# export sent_dir=”$HCISITEDIR/data/golive_archive_resent”
# export sent_log=$sent_dir/.log
# export kill_file=”$que_dir/.kill”
# export lock_file=”$que_dir/.lock”
# export rs_proc=”resend_rxtfc_adt_9027″
# export load_thread=”load_rxtfc_adt_9027″
# export rs_thread=”rs_rxtfc_adt_9027″
# resend_que.ksh
#
# Be sure to make an entry in /hci/cron/reminders.ksh as long as this script is scheduled to run via cron
#
#———
# History:
#———
#
# 2004.04.28 Russ Ross
# – wrote intial version
#
#
# 2004.04.30 Russ Ross
# – modified to only recycle the load thread instead of recycling the process
# – modified so this script will not run unless the following conditions are true:
#
# the resend process is running
# the resend thread is UP
#
# End of Module Header =============================================================================#—————————————-
# make sure the resend process is running
#—————————————-if [ ! -f $HCISITEDIR/exec/processes/$rs_proc/pid ]; then
echo “”
echo ” STOPPING because the resend process is not running”
echo ” $rs_proc”
echo “”
exit
fi#———————————————–
# make sure the resend thread is UP=2 or INEOF=6
#———————————————–proto_status=`hcimsiutil -dd $rs_thread | grep “^Proto Status” | awk -F: ‘{print $2}’ | tr -d ‘ ‘`
if [[ “$proto_status” != “2” && “$proto_status” != “6” ]]; then
echo “”
echo ” STOPPING because the resend thread is not UP or INEOF”
echo ” $rs_thread”
echo “”
exit
fi#—————————————-
# check if the necessary directories exit
#—————————————-if [ ! -d $que_dir ]; then
echo “”
echo ” FATAL ERROR – directory does not exist”
echo ” $que_dir”
echo “”
exit
fiif [ ! -d $load_dir ]; then
echo “”
echo ” FATAL ERROR – directory does not exist”
echo ” $load_dir”
echo “”
exit
fiif [ ! -d $sent_dir ]; then
echo “”
echo ” FATAL ERROR – directory does not exist”
echo ” $sent_dir”
echo “”
exit
fi#——————————————
# check if this script has been toggled off
#——————————————if [ -f $kill_file ]; then
echo “”
echo ” STOPPING because this script has been toggled off”
echo ” $kill_file”
echo “”
exit
fi#—————————————-
# check if this script is already running
#—————————————-if [ -f $lock_file ]; then
echo “”
echo ” STOPPING because this script is already running”
echo ” $lock_file”
echo “”
exit
fi#————————————————————
# create lock file to prevent more than one occurance running
#————————————————————touch $lock_file
#————————————-
# make sure the last load file is done
#————————————-if [ -f $load_dir/* ]; then
echo “”
echo ” STOPPING because not done resending last load file in directory”
echo ” $load_dir”
echo “”
rm -f $lock_file
exit
fi#—————————————————————————
# make sure that no outbound messages are queued up over the next 60 seconds
#—————————————————————————let count=1
while ((count <= 7)); do ob_que_depth=`hcimsiutil -dd $rs_thread | grep "^OB Data QD" | awk -F: '{print $2}' | tr -d ' '` if [[ $ob_que_depth > 0 ]]; then
echo “”
echo ” STOPPING because outbound messages are still queued up”
echo “”
rm -f $lock_file
exit
fisleep 10
let count=count+1done
#————————————-
# stop thread if it is UP=2 or INEOF=6
#————————————-proto_status=`hcimsiutil -dd $load_thread | grep “^Proto Status” | awk -F: ‘{print $2}’ | tr -d ‘ ‘`
if [[ “$proto_status” = “2” || “$proto_status” = “6” ]]; then
hcicmd -p $rs_proc -c “$load_thread pstop”
sleep 5
fi#——————————————-
# zero out thread counts for the load thread
#——————————————-hcimsiutil -zt $load_thread >/dev/null
sleep 5#—————————————-
# get the name of the next file to resend
#—————————————-resend_file=””
if [ -f $que_dir/* ]; then
resend_file=`cd $que_dir; ls -1 * | sort | head -1`
fiif [[ “$resend_file” = “” ]]; then
echo “resend_que.ksh – no file found”
rm -f $lock_file
exit
fi#——————————–
# make entry into resend log file
#——————————–dts=`date “+%Y.%m.%d_%H:%M:%S”`
echo “$dts resending $resend_file” >> $sent_log#—————————————————–
# copy the next file to resend into the load directory
#—————————————————–cp -p $que_dir/$resend_file $load_dir
#——————————————————-
# move the next file to resend into the resent directory
#——————————————————-mv $que_dir/$resend_file $sent_dir
#———————-
# start the load thread
#———————-hcicmd -p $rs_proc -c “$load_thread pstart”
sleep 5#———————
# get rid of lock file
#———————rm -f $lock_file
Here is an example of a calling script ( rkr_xcelera_resend.ksh ):
Code:#!/usr/bin/ksh
# Begin Module Header ==============================================================================
#
#
#——
# Name:
#——
#
# rkr_xcelera_resend.ksh
#
#———
# Purpose:
#———
#
# export/set the variables that get used by the generic script resend_que.ksh
#
# will resend the next que file in name sorted order
#
# if you schedule this in cron to run every 10 minutes
# it will resend each que file one by one in name sorted order
# by cycling the resend process and thread counts for each file sent before proceeding to the next file
# plus will perpetually restartied every 10 minutes until all files have been resent
# or until the kill_file has been touched
#
#——-
# Notes:
#——-
#
# Example of how to schedule this to run every minute in cron:
#
# 00-59 * * * * /bin/ksh -c ‘eval . ~/.profile.cron /quovadx/qdx5.2/integrator test_xcelera; rkr_xcelera_resend.ksh > /dev/null’
#
# Be sure to make an entry in /hci/cron/reminders.ksh as long as this script is scheduled to run via cron
#
#———
# History:
#———
#
# 2006.11.09 Russ Ross
# – wrote intial version
#
# End of Module Header =============================================================================#—————————————————————————–
# directory of the name sorted files to be sent
# (hidden file like .kill or .lock will not be a part of the sorted name list)
#—————————————————————————–export que_dir=”$HCISITEDIR/data/que_xcelera_adt”
#————————————————————————
# directory to move the next name sorted file to one by one for resending
#————————————————————————export load_dir=”$HCISITEDIR/data/load_xcelera_adt”
#—————————————————
# directory that will hold the files that got resent
#—————————————————export sent_dir=”$HCISITEDIR/data/sent_xcelera_adt”
#—————————-
# log file of what was resent
#—————————-export sent_log=$sent_dir/.log
#————————————————————————————————–
# toggle file that can be touched to gracefully prevent future invoking of this script from running
#————————————————————————————————–export kill_file=”$que_dir/.kill”
#—————————————————————————–
# file to prevent multiple invokations of this script running at the same time
#—————————————————————————–export lock_file=”$que_dir/.lock”
#—————————
# name of the resend process
#—————————export rs_proc=”resend_xcelera_adt”
#————————
# name of the load thread
#————————export load_thread=”load_xcelera_adt”
#————————–
# name of the resend thread
#————————–export rs_thread=”ob_xcelera_adt”
#—————————————-
# pass these arguements to resend_que.ksh
#—————————————-# resend_que.ksh >/dev/null
resend_que.ksh
Russ Ross
RussRoss318@gmail.com
-
- The forum ‘Cloverleaf’ is closed to new topics and replies.
