Cloverleaf high availability crontab synchronization

Homepage Clovertech Forums Read Only Archives Cloverleaf Operating Systems Cloverleaf high availability crontab synchronization

  • Creator
    Topic
  • #53639
    Christopher Wells
    Participant

    We want to eliminate the process of manually synchronizing crontab for Cloverleaf-related jobs when an active node fails over to the inactive node.

    Christopher Wells

Viewing 3 reply threads
  • Author
    Replies
    • #78388
      James Cobane
      Participant

      Christopher,

      I don’t see an attachment.

      Jim Cobane

      Henry Ford Health

    • #78389
      Christopher Wells
      Participant

      Thanks for mentioning the attachment didn’t post properly.  I’ve updated the original post.

      Christopher Wells

    • #78390
      Russ Ross
      Participant

      I’m going to share how I have dealt with crontab concerns on HA fail-over instead of advising you on your approach.

      If you decide to read my approach, you will notice it does not require any synchronization and I can vouch it has proven effective thru sever OS and cloverleaf and HA upgrades over a 15 year period.

      Frist create the one and only controlling file for hci cron entires somewhere on the disk space that is mounted to the active node by doing this

      Code:

      crontab -l >/somewhere/hci.cron

      NOTE: the /somwhere directory is expected to be mounted on which ever node is active but not the passive node and I suspect your HA fail-over already does this.

      Second to make any new updates to hci cron entries modify hci.cron on the active node as follows:

      Code:

      cd /somewhere
      cp -p hci.cron hci.cron_2013.05.23_russ
      vi hci.cron
      crontab hci.cron

      Third modify HA shutdown script to do the following to unload active hci cron entries on shutdown:

      Code:

      su hci -c ‘crontab -r’

      Fourth modify HA start up script to do the following to load cron entries upon startup:

      Code:

      su hci -c ‘crontab /somewhere/hci.cron’

      As you can see with this method there is only one controlling file so nothing to keep in sync and it is only loaded on the currently active node.

      You also have backup copies if you need to revert back or recover to a previous version of hci.cron.

      Russ Ross
      RussRoss318@gmail.com

    • #78391
      Eric Fortenberry
      Participant

      I work with Christopher, and today we failed over our test system using the crontab syncronization scripts mentioned above.  Everything appeared to work great.  The only complaint that I have is that on our system (AIX 6.1), when we load crontab from a file, we lose all the blank lines that help separate and format our different cron jobs.

      One change that I have since made to this process is to have the inactive crontab entry make a copy before loading the active crontab.  This should provide us with some information about the state of the crontabs whenever failover occurs.

      So, with all that being said, here is the inactive crontab that we now have:

      Code:

      # This is the inactive version of a crontab that is loaded on
      # the inactive node of the Cloverleaf cluster.
      #    DO NOT DELETE THIS CRON ENTRY OR FILE!!!!!!

      # If the active crontab is available (and thus this node is active), load the active crontab.
      * * * * * test -f /clovermisc/cronsync/crontab.active && cp -p /clovermisc/cronsync/crontab.active /clovermisc/cronsync/crontab.loaded.$(date “+%Y%m%d%H%M%S”) && crontab /clovermisc/cronsync/crontab.active

      And here is the important pieces of the active crontab:

      Code:

      #########################################
      # Crontab syncing b/w cluster nodes
      #########################################
      # Load the inactive crontab if the active crontab can’t be found
      * * * * * test -f /clovermisc/cronsync/crontab.active || crontab ~/crontab.inactive

      # Dump the active crontab in case of change
      * * * * * test -d /clovermisc/cronsync && crontab -l > /clovermisc/cronsync/crontab.active

      As you see, we dump the currently active crontab.  This eliminates the need for any “unusual” steps to update the crontab.  We can just run “crontab -e” to edit; then on failover, the changes are “just there”.

      Thanks,

      Eric

Viewing 3 reply threads
  • The forum ‘Operating Systems’ is closed to new topics and replies.

Forum Statistics

Registered Users
5,129
Forums
28
Topics
9,301
Replies
34,447
Topic Tags
288
Empty Topic Tags
10