Cloverleaf high availability crontab synchronization

[Christopher Wells]

We want to eliminate the process of manually synchronizing crontab for Cloverleaf-related jobs when an active node fails over to the inactive node.

Christopher Wells

Topic

[James Cobane]

Christopher,

I don’t see an attachment.

Jim Cobane

Henry Ford Health

[Christopher Wells]

Thanks for mentioning the attachment didn’t post properly.  I’ve updated the original post.

Christopher Wells

[Russ Ross]

I’m going to share how I have dealt with crontab concerns on HA fail-over instead of advising you on your approach.

If you decide to read my approach, you will notice it does not require any synchronization and I can vouch it has proven effective thru sever OS and cloverleaf and HA upgrades over a 15 year period.

Frist create the one and only controlling file for hci cron entires somewhere on the disk space that is mounted to the active node by doing this

Code:

crontab -l >/somewhere/hci.cron

NOTE: the /somwhere directory is expected to be mounted on which ever node is active but not the passive node and I suspect your HA fail-over already does this.

Second to make any new updates to hci cron entries modify hci.cron on the active node as follows:

Code:

cd /somewhere
cp -p hci.cron hci.cron_2013.05.23_russ
vi hci.cron
crontab hci.cron

Third modify HA shutdown script to do the following to unload active hci cron entries on shutdown:

Code:

su hci -c ‘crontab -r’

Fourth modify HA start up script to do the following to load cron entries upon startup:

Code:

su hci -c ‘crontab /somewhere/hci.cron’

As you can see with this method there is only one controlling file so nothing to keep in sync and it is only loaded on the currently active node.

You also have backup copies if you need to revert back or recover to a previous version of hci.cron.

Russ Ross
RussRoss318@gmail.com

[Eric Fortenberry]

I work with Christopher, and today we failed over our test system using the crontab syncronization scripts mentioned above.  Everything appeared to work great.  The only complaint that I have is that on our system (AIX 6.1), when we load crontab from a file, we lose all the blank lines that help separate and format our different cron jobs.

One change that I have since made to this process is to have the inactive crontab entry make a copy before loading the active crontab.  This should provide us with some information about the state of the crontabs whenever failover occurs.

So, with all that being said, here is the inactive crontab that we now have:

Code:

# This is the inactive version of a crontab that is loaded on
# the inactive node of the Cloverleaf cluster.
#    DO NOT DELETE THIS CRON ENTRY OR FILE!!!!!!

# If the active crontab is available (and thus this node is active), load the active crontab.
* * * * * test -f /clovermisc/cronsync/crontab.active && cp -p /clovermisc/cronsync/crontab.active /clovermisc/cronsync/crontab.loaded.$(date “+%Y%m%d%H%M%S”) && crontab /clovermisc/cronsync/crontab.active

And here is the important pieces of the active crontab:

Code:

#########################################
# Crontab syncing b/w cluster nodes
#########################################
# Load the inactive crontab if the active crontab can’t be found
* * * * * test -f /clovermisc/cronsync/crontab.active || crontab ~/crontab.inactive

# Dump the active crontab in case of change
* * * * * test -d /clovermisc/cronsync && crontab -l > /clovermisc/cronsync/crontab.active

As you see, we dump the currently active crontab.  This eliminates the need for any “unusual” steps to update the crontab.  We can just run “crontab -e” to edit; then on failover, the changes are “just there”.

Thanks,

Eric

[Author]

Replies