Backup of a running cloverleaf site

[Linus Betschart]

Hi Community

I want to regulary backup a specific site within the cloverleaf instance. Just to copy away the files from C:cloverleafcis6.1integrator[my site] is not an option as when Cloverleaf is running some files are locked. Furthermore, I think there might be an inbuilt option in Cloverleaf to automate this task?

Unfortunately, I have not found anything helpful in the documentation.

How can I backup a site from a running cloverleaf instance away? with all configurations?

Your help is much appreciated!!

Regards,

Linus

Topic

[Elisha Gould]

Are you just wanting to backup the configuration?

hcibox was introduced in 6.x that will grab the configuration and save a snapshot of the site.

It will save the configuration for the site in $HCIROOT/box and can save a zipped up box file.

ie the following will save the config to $HCIROOT/box

hcibox -s existing_site existing_site

the following will zip up the config

hcibox -e /tmp/backup_box existing_site

[Linus Betschart]

Hi Elisha

thank you for your response!

I saw this command initially, but the -s switch you are using does not exist in my version (6.1P).

the most similar switch would be -n, but the disadvantage is, you have to reference the threads you want to backup. I need a general script which can backup any site without knowing the containing threads.

working example when knowing the containing threads to save the config to $HCIROOT/box:

Code:

hcibox -n siteXY THREAD1,THREAD2,THREAD3 r siteXY_box

Furthermore, I cannot add additional resources to this box via scripting. Can I?

[Max Drown (Infor)]

You can safely skip the locked files when copying. These are most likely the shared memory and runtime files which are not needed in the backup.

-- Max Drown (Infor)

[Russ Ross]

I have found this comment from Max to allow me to ignore the locked files that can’t be copied and still get a useable backup of an entire site.

Code:

You can safely skip the locked files when copying. These are most likely the shared memory and runtime files which are not needed in the backup.

I have a couple of more elaborate scripts (tar_site_to_move.ksh & untar_prep_site.ksh) that are warppers around a script that does the actual site copies called tar_tar_dir.ksh seen here:

#!/usr/bin/ksh

# Begin Module Header ==============================================================================
#
#——
# Name:
#——
#
# tar_tar_dir
#
#———
# Purpose:
#———
#
# Copy an entire directory tree with all files and links to another base directory
# using the tar command
#
# Notes:
#——-
#
# This will not traverse logical links across file systems,
# so everything copied has to reside on the same file system.
#
#
# This script is nice because it pipes the output of one tar command to another,
# which requires a little bit of temporary memory.
#
# Example of normal usage:
#
# cd $HCIROOT
# tar_tar_dir ./test_picis /3.5.2P_picis
#
#———
# History:
#———
#
# 1999.10.29 Russ Ross
# [code]#!/usr/bin/ksh

# Begin Module Header ==============================================================================
#
#

---

# Name:
#

---

#
# tar_tar_dir
#
#

---

# Purpose:
#

---

#
# Copy an entire directory tree with all files and links to another base directory
# using the tar command
#
# Notes:
#

---

#
# This will not traverse logical links across file systems,
# so everything copied has to reside on the same file system.
#
#
# This script is nice because it pipes the output of one tar command to another,
# which requires a little bit of temporary memory.
#
# Example of normal usage:
#
# cd $HCIROOT
# tar_tar_dir ./test_picis /3.5.2P_picis
#
#

---

# History:
#

---

#
# 1999.10.29 Russ Ross
#

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Here is the script to copy a site to be cloned.

Even though its called tar_site_to_move.ksh it isn’t required to move the site to another box and can also be used to clone the site on the existing box.

I have used it to created cloned versions of an existing site whne making extensive and time consuming modifications to a site so the running site stays intact until I’m done with the new version modifications and can flip the symbolic link for the site between version_1 and version_2 as desired and once ready have site version_2 replace site version_1.

Code:

#!/usr/bin/ksh

clear
echo “”
showroot
echo “”
echo “Is this the site you want to move to another machine?”
echo “(y to continue; anything else to quit)”
read xyz
if [ “$xyz” != “y” ]; then
    exit
fi
if [[ ! -d /upgrade ]]; then
    echo “FATAL ERROR – need to manually create /upgrade directory”
    exit
fi
if [[ ! -d /upgrade/new_box ]]; then
    echo “FATAL ERROR – need to manually create /upgrade/new_box directory”
    exit
fi
cd /upgrade/new_box
rm -rf /upgrade/new_box/$HCISITE /upgrade/new_box/$HCISITE.tar
mkdir /upgrade/new_box/$HCISITE
tar_tar_dir.ksh $HCISITEDIR /upgrade/new_box/$HCISITE
tar -cvf $HCISITE.tar ./$HCISITE
rm -rf /upgrade/new_box/$HCISITE

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Here is the remaining counterpart script untar_prep_site.ksh that will deposit the site to move in the desired place specified.

I posted to illustrate the preprocessing and removal of files I do before using a cloned site.

Our architecture will different than yours so the script will not likely run as is at your enterprise, but it illustrates the considerations and choices I made for prepping a cloned copy of a site.

These scripts predate the BOX capability so now I use both methods depending on which one gets what I need done best.

Code:

#!/usr/bin/ksh

# Begin Module Header ==============================================================================
#
#——
# Name:
#——
#
# untar_prep_site.ksh
#
#———
# Purpose:
#———
#
# Use the tar file from running tar_site_to_move.ksh to create a new site.
#
#
#——–
# Inputs:
#——–
#
# $1 = name of the original site used to create the tar_site_to_move
# $2 = name of the new site to create
#
#——-
# Notes:
#——-
#
# Here is an example of normal usage of this script
#
#     untar_prep_site.ksh   test_sms_genie   t_sms_genie
#
# Using this script will take care of many of the tasks associated with creating a new site at MDACC,
#
# This script along with tar_site_to_move.ksh is especially usefull for any of the following:
#
# – create a new site that is a copy of an existing site
# – move a site from one machine to another
#
#———
# History:
#———
#
# 2009.01.12 Russ Ross
#          – wrote initial version
#            by converting the hardcoded version he had used during cloverleaf upgrades for many years.
#
# 2012.12.10 Russ Ross
#          – added logic to clean out site_startup_backups directory
#
# 2015.02.17 Russ Ross
#          – added clean up logic to get rid of
#            #MDA_UPDIR/new_box/$site_name/scripts/site_shutdown.ksh
#            $MDA_UPDIR/new_box/$site_name/scripts/site_startup.ksh

# End of Module Header =============================================================================

#———————————————————-
# set the MDA_UPDIR environment variable if not already set
#———————————————————-

if [[ “`echo $MDA_UPDIR`” = “” ]]; then
   export MDA_UPDIR=/upgrade
fi

#———————————————
# make sure this script is run by the hci user
#———————————————

me=`whoami`

if [ “$me” != “hci” ]; then
 echo “”
 echo ”     $0 must be run as hci”
 echo “”
 exit 1
fi

#———————————————-
# check if old site name was give as argument 1
#———————————————-

site_name=”$1″
if [ “$site_name” = “” ]; then
   echo “”
   echo “***** ERROR – must supply OLD site name *****”
   echo “”
   echo “Here is one example of normal usage of this scripts:”
   echo “”
   echo ”     untar_prep_site.ksh   test_sms_genie   t_sms_genie”
   echo “”
   exit 2
fi

#——————————————————
# see if a tar file exists for the old site to be moved
#——————————————————

cd $MDA_UPDIR/new_box
if [[ ! -a $site_name.tar ]]; then
   echo “”
   echo “***** ERROR – no file called ( $site_name.tar ) in directory ( $MDA_UPDIR/new_box ) *****”
   echo “”
   echo “Here is what is in directory ( `pwd` )”
   echo “”
   ls -al
   echo “”
   exit 3
fi

#—————————————————
# check if the new site name was given as argument 2
#—————————————————

new_site=”$2″
if [ “$new_site” = “” ]; then
   echo “”
   echo “***** ERROR – must supply NEW site name *****”
   echo “”
   echo “——————————————–”
   echo “Here are some of the possible site prefixes:”
   echo “——————————————–”
   echo “”
   echo ”     mda_global”
   echo ”     p_mock_”
   echo ”     p_new_”
   echo ”     p_old_”
   echo ”     p_real_”
   echo ”     prod_batch”
   echo ‘     prod_batch_###’
   echo ”     test_upgrade”
   echo “”
   echo ”     mda_global”
   echo ”     t_”
   echo ”     test_batch”
   echo ‘     test_batch_###’
   echo ”     test_upgrade”
   echo “”
   echo “Here is one example of normal usage of this scripts:”
   echo “”
   echo ”     untar_prep_site.ksh   test_sms_genie   t_sms_genie”
   echo “”
   exit 4
fi

#—————————————————–
# check to be sure the new site does not already exist
#—————————————————–

### if [[ -d $HCIROOT/$new_site ]]; then
if [ “`ls $HCIROOT/$new_site 2>/dev/null`” ]; then
   echo “”
   echo “***** ERROR – the new site ( $new_site ) already exist”
   echo “***** you will need to rename or delete it and rerun this script again”
   echo “***** this script is terminating without taking any action”
   echo “”
   exit 5
fi

#——————————————
# create the new site and update server.ini
#——————————————

### if [[ ! -d $HCIROOT/$new_site ]]; then
if [ ! “`ls $HCIROOT/$new_site 2>/dev/null`” ]; then
    echo “”
    echo “==============================”
    echo “”
    echo ”     About to do ( hcisiteinit   $new_site )”
    echo ”     respond y to accept”
    echo ”     any other response will abort this script”
    echo “”
    read response
    if [[ “$response” != “y” ]]; then
        echo “”
        echo “This script has been terminated without doing any action.”
        echo “”
        exit 6
    fi
    echo “”
    echo ” creating new site ( $new_site )”
    echo “”
    hcisiteinit $new_site
    set_server_ini_environs.ksh
    if [[ ! -d $HCIROOT/$new_site ]]; then
        echo “”
        echo “***** ERROR – hcisiteinit did not create directory ( $HCIROOT/$new_site ) *****”
        echo “”
        exit 7
    fi
fi

#————————————————
# replace the site directory with a symbolic link
#————————————————

if [[ ! -d /sites ]]; then
   echo “ERROR will need to create /sites file system”
   exit 9
fi
if [[ ! -d /sites/$HCIVERSION ]]; then
   mkdir /sites/$HCIVERSION
fi
if [[   -d /sites/$HCIVERSION/$new_site ]]; then
   echo “ERROR /sites/$HCIVERSION/$new_site already exists”
   exit 9
fi
if [[ ! -d /sites/$HCIVERSION/$new_site ]]; then
   mkdir /sites/$HCIVERSION/$new_site
   mkdir /sites/$HCIVERSION/${new_site}/`date +”v01_%Y.%m.%d”`
fi

tar_tar_dir.ksh $HCIROOT/$new_site
           /sites/$HCIVERSION/${new_site}/`date +”v01_%Y.%m.%d”`

rm -rf $HCIROOT/$new_site

ln -s /sites/$HCIVERSION/${new_site}/`date +”v01_%Y.%m.%d”` $HCIROOT/$new_site

#——————-
# untar the old site
#——————-

cd $MDA_UPDIR/new_box
rm -rf $site_name
tar -xvf $site_name.tar

if [[ ! -d $MDA_UPDIR/new_box/$site_name ]]; then
   echo “”
   echo “***** ERROR – tar extract did not create directory ( $MDA_UPDIR/new_box/$site_name ) *****”
   echo “”
   exit 8
fi

#—————————————————————————
# clean up the old site in preparation of using it to overwrite the new site
#—————————————————————————

cd $MDA_UPDIR/new_box/$site_name
rm -rf `ls -al | grep “^-” | awk ‘{print $9}’ | grep -v “^NetConfig$”`
rm -rf exec
rm -rf lock; mkdir lock
rm -rf logs; mkdir logs
rm -rf revisions; mkdir revisions
rm -rf stats; mkdir stats
rm -f  ./Alerts/qdx*_hide_default.alrt

#—————————
# get rid of all *.bak files
#—————————

cd $MDA_UPDIR/new_box/$site_name
find . -name “*.bak” -exec rm -rf {} ;
find . -name “backup” -exec rm -rf {} ;
find . -name “x.x” -exec rm -rf {} ;
find . -name “core” -exec rm -rf {} ;

#————————————————————————-
# make sure the site specific tclprocs directory only contains *.tcl files
#————————————————————————-

cd $MDA_UPDIR/new_box/$site_name/tclprocs
rm -rf `ls -l | grep -v “^-” | awk ‘{print $9}’`
rm -f `ls -1 | grep -v “.tcl$”`
rm -f .*.tcl

#———————————————————————-
# make sure the site specific Xlate directory only contains *.xlt files
#———————————————————————-

cd $MDA_UPDIR/new_box/$site_name/Xlate
rm -rf `ls -l | grep -v “^-” | awk ‘{print $9}’`
rm -f `ls -1 | grep -v “.xlt$”`
rm -f .*.xlt

#———————————————
# clean out the site_startup_backups directory
#———————————————

rm -rf $MDA_UPDIR/new_box/$site_name/scripts/site_shutdown.ksh
rm -rf $MDA_UPDIR/new_box/$site_name/scripts/site_startup.ksh
rm -rf $MDA_UPDIR/new_box/$site_name/scripts/site_startup_backups/*

#————————————————–
# set the ownership of the entire site to hci:staff
#————————————————–

chown -R hci:staff $MDA_UPDIR/new_box/$site_name

#—————————————–
# tar_tar_dir the old_site to the new_site
#—————————————–

if [[ -d $HCIROOT/$new_site ]]; then
    echo “”
    echo “==============================”
    echo “”
    echo ”     About to do ( tar_tar_dir   $MDA_UPDIR/new_box/$site_name   $HCIROOT/$new_site )”
    echo ”     respond y to accept”
    echo ”     any other response will abort this script”
    echo “”
    read response
    if [[ “$response” != “y” ]]; then
        echo “”
        echo “This script has been terminated without doing.”
        echo “”
        echo ”     tar_tar_dir   $MDA_UPDIR/new_box/$site_name   $HCIROOT/$new_site ”
        echo “”
        exit 9
    fi
    tar_tar_dir.ksh   $MDA_UPDIR/new_box/$site_name   $HCIROOT/$new_site
fi

#——————————————————–
# create the process directories defined in the NetConfig
#——————————————————–

echo “”
echo “creating the process directories”
procList=`grep “^process” $HCIROOT/$new_site/NetConfig | awk ‘{print $2}’ | sort`
cd $HCIROOT/$new_site/exec/processes
for proc in $procList; do
   rm -rf $proc; mkdir $proc
done
ls -al
echo “”

#——————–
# link data directory
#——————–

   echo “”
   echo “—————”
   echo “ln -s   /data/$new_site   $HCIROOT/$new_site/data”
   echo “—————”
   echo “”
   if [[ ! -a /data/$new_site ]] then
       echo “”
       echo “creating /data/$new_site directory to link too”
       echo “”
       mkdir /data/$new_site
       ln -sf /data/$new_site $HCIROOT/$new_site/data
   else
       if [[ -a $HCIROOT/$new_site/data ]] then rm -rf $HCIROOT/$new_site/data; fi
       ln -sf /data/$new_site $HCIROOT/$new_site/data
   fi

#—————
# make TCL index
#—————

       echo “”
       echo “Doing a mktclindex”
       cd $HCIROOT/$new_site/tclprocs
       mktclindex
       ls -al
       echo “”

#————————————————
# clear database for safety
# and reset shared memory pointer/file for safety
#————————————————

echo “”
echo “About to clear data base for safety”

myRoot=$HCIROOT
mySite=$SITE

setroot -clear
setroot $myRoot $new_site

showroot
clear_db.ksh

cd $HCIROOT/$new_site/exec
rm -f monitorShmemFile
hcimsiutil -R

setroot -clear
setroot $myRoot $mySite

#——————————————-
# hide Alerts and clean out Alerts directory
#——————————————-

echo “Hiding alerts and cleaning out Alerts directory”
if [[ ! -d $HCIROOT/$new_site/Alerts ]] then
   echo “ERROR – no Alerts direcotry exists”
   exit
fi

cd $HCIROOT/$new_site/Alerts
cp -p default.alrt .hide_original__default.alrt
rm -rf *
cp -p .hide_original__default.alrt hide_default.alrt
cp $HCIROOT/siteProto/Alerts/off.alrt .
ln -sf off.alrt default.alrt

ls -al
echo “”

#—————————————————-
# reminder to manually create the data subdirectories
#—————————————————-

echo “”
echo “************”
echo “IMPORTANT — you will have to manually create the /data/$new_site subdirectories”
echo “ALSO; do a save in the GUI NetConfig to see if any errors show up in addition to doing a netcheck”
echo “ALSO; a symbolic link for default.alrt has been set to point to off.alrt”
echo “************”
echo “”

Russ Ross
RussRoss318@gmail.com

[Linus Betschart]

thank you so much for your responses. awesome!!

this is what I scripted and have in use right now:

Code:

@echo off
cls
rem
rem
rem Batch to backup interface site data to the network$ folder
rem
rem
rem     %1  –   source (ex. C:cloverleafcis6.1integrator)
rem     %2  –   site name (abcdefg)
rem     %3  –   destination (ex. \servershare$)
rem     %4  –   processes (ex. “ADT,DFT,SIU”)
rem    
rem ———————————————————————
rem  Version    Date           Init       Changes
rem ———————————————————————
rem  1.0.00     06/28/2016     lbetscha   Initial Version
rem ———————————————————————

setlocal
set LOG_FILE=%~dp0robo_log.log
set BACKUP_LOG_FILE=%~dp0backup.log

rem ———————————————————————
echo.
echo ****************************************
echo Set Site…
set site=%2
echo Site                    – %site%
echo Site                    – %site% > %BACKUP_LOG_FILE%

rem ———————————————————————
echo.
echo ****************************************
echo Set Source…
set source=%1%site%
echo Backup from             – %source%
echo Backup from             – %source% >> %BACKUP_LOG_FILE%

rem ———————————————————————
echo.
echo ****************************************
echo Create Backup Folder…
set backupfolder=%3interface%site%
echo Backup destination – %backupfolder%
echo Backup destination – %backupfolder% >> %BACKUP_LOG_FILE%

md %backupfolder%

rem ———————————————————————
echo.
echo ****************************************
echo Set Processes…
echo Processes               – %4
echo Processes               – %4 >> %BACKUP_LOG_FILE%

rem ———————————————————————
echo.
echo ****************************************
echo Set cloverleaf environment variables
CALL d:cloverleafcis6.1integratorsbinsetroot.cmd
CALL d:cloverleafcis6.1integratorsbinshowroot.cmd

rem ———————————————————————
echo.
echo ****************************************
echo Stop cloverleaf processes
hcienginestop -p %4

rem ———————————————————————
echo.
echo ****************************************
echo Starte Sicherung…
robocopy %source% %backupfolder% /MIR /B /XD “%source%exec” /E /COPY:DAT /LOG:%LOG_FILE% /R:10 /W:30

echo.
echo Sicherung abgeschlossen.

rem ———————————————————————
echo.
echo ****************************************
echo Clean up cloverleaf processes…
hcilmclear -p %4
echo.
echo Clean up abgeschlossen.

echo.
echo ****************************************
echo Start cloverleaf processes…
hcienginerun -p %4

[Author]

Replies