Revision/Change Control Methods?

[Jennifer Hardesty]

We are in the process of standardizing our Cloverleaf revision/change control methodology and policy.

What I’ve learned from the process is that there is no right or wrong way, but there are a lot of different ways and everyone has their own thoughts on how it should be done.

Does your facility have a policy/method in place?  What is it?  Are you automated or do you work with checklists?  Do you rely on using the Notes section of threads and the revisions folder in Cloverleaf?  Do you have Excel Spreadsheets or do you have some fancy software tracking package?  

Thank you for sharing your information ahead of time.

Topic

[Russ Ross]

A quick description is that we use a checcklist that I created long ago before things like master sites or Notes directories even existed or I even knew there were revisons directories.

We use a symbolic link at the HCIROOT level to point to the active version of a given site.

When ready to implement a change to the currently active version of a given site, we use our checklist to clone the active site to create a new version of that site.

Using a new temproary link to the cloned site we make our changes and mock test, then we are ready to golive.

To go live we shutdown the active site and point the link from the shutdown site to the new cloned/modified site and start the site up.

This way current production site version is left in tact and if we need to back out from new production site version we have 100% confidence we can restore to what we had by switching the link back to the previous site version.

This saves you from any and all problems you might run into while modifying the new site version knowing you can always get back to your starting point, plus you can test incrmentally changes in the PROD environment of the newly cloned site before crossing your fingers when you turn them on live.

If you want to lower stress and lower risk this approach is design with that intent in mind.

There are other approaches that can be less work but I put the emphasis on lowering risk and stress to increase the number of successful golives and the ability to back out flawlessly without question or much thought about what needs to be done to be back to a known working version of the site.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Along with what I mentioned, we have many sites that are relatively small and more granular.

This architecture augments nicely with our cloned sites golive approach for PROD changes.

The combination of the smaller granular sites along with cloned site versioning, gives us options to greatly control the impact of changes to PROD to a reasonable level and not necessarily all at once.

Russ Ross
RussRoss318@gmail.com

[Jennifer Hardesty]

Russ,

What you describe would work fine for being able to back out of any issues with a NetConfig where a route has changed or a pre-proc has been added to a route or even a port has been altered , but what about restoring a Xlate, tclproc, variant, Table, or even an alert?

🙂

[Jim Kosloskey]

Jennifer,

The Revisions direectory does not back up any of the Message Formats such as HL/7, X12, XML, etc. It only contains backups for FRL, HRL, and VRL.

The Revisions directory also does not contain any backup for tclprocs unless you use the Tcl editor through the IDE. If you use vi or Ultra-Edit, or PS-Pad or any other more full function language editors then you will have to provide your own backups.

Also the naming convntion for the Revisions files could be done better.

I know some peoiple claim to have had some success with various code management products but I am not sure if they will provide for handling all of the Cloverleaf objects.

You might need to end up with a number of tools including something as simple as making a backup copy of an object via O/S commands in use in order to get what you need.

email: jim.kosloskey@jim-kosloskey.com 29+ years Cloverleaf, 59 years IT - old fart.

[Jennifer Hardesty]

Well, that’s why I was asking what methods folks use.  What keeps you protected in those emergencies when you have to back everything out?  Do you have a manual system in place and if so, what does it involve?

I’m just curious.

[James Cobane]

Typically, we will backup all of the files that we are updating by putting them into .tar file; i.e. at the site level, create a .tar file and add all the configurations you are changing to it:

tar -cvf myBackup.tar Xlate/myxlate.xlt

tar -uvf myBackup.tar formats/hl7/2.3/myVariant

tar -uvf myBackup.tar tclprocs/myproc

..etc

Then you can re-extract the .tar file to quickly restore the old versions.  For NetConfig changes, you should create a backup by performing a ‘Save As’ to some backup name.  Then to restore it you would open the backup you created and ‘Save As’ NetConfig.  You need to do the ‘Save As’ to retain the notes information….

Jim Cobane

Henry Ford Health

[Russ Ross]

Code:

Russ,

What you describe would work fine for being able to back out of any issues with a NetConfig where a route has changed or a pre-proc has been added to a route or even a port has been altered , but what about restoring a Xlate, tclproc, variant, Table, or even an alert?

It takes care of everything in the site because you never change the running site, you only change the cloned site that will replace the currently running site.

When I say clone of the site, I meant the entire site is cloned down to the last byte and directory in that site when the snapshot is taken.

This method also saves you from even having to remember what to back out because you just reset the site link to point back to the original site that was never changed.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Maybe this will help to visualize, which is not as high level but not our lengthy checklist that we usually mentor someone on at least 3 times before expecting them to gain any self reliance.

Russ Ross
RussRoss318@gmail.com

[Jennifer Hardesty]

I see.  I suppose that method does work well if you

A.) Are not using a master site method where all of your tclprocs, Xlates, etc. are in one global location.

B.) Have unlimited space to have unlimited copies of every file in every site, including the smat files for however long you archive them — some of our sites keep smats for up to 90 days.

[Russ Ross]

We have a master site and it can be versioned right along with any other site if desired.

This reminds me I have a wish list item that also includes doing tar backups of configuration files each day across all sites that can be restored down to a single file.

We typically put something at a local site level for a while before promoting it to the master site.

For some of the simpler implemenations we would be pardoned from doing a site clone and just make a backup of the single file that is changing.

This tends to occur more often in the master site since there are only individul objects and no running interfaces what so ever.

Unlimited space would be a concern if an entire site took up much room, which it doesn’t.

I checked and all of our sites combined in prodcution and their site clones take up less than 2GBs.

Having said that we have /work and /ftp and /data and /archive file systems outside our sites so the sites only have in them mostly what they need to exist and run.

We also cycle SMAT every 3 hours during the day and archive it due to our 6 million message flow a day, which keeps sites from growing overly large.

We keep 9 months to one year of SMAT in our /archive file systems but not in our /sites file system.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Now that you got me thinking about the master site there is another consideration.

Once you change a global object it is required to make sure everything that uses it gets refreshed to pick up the change.

A component of our IPL/HA scripts comes into play at this point.

The way we have designed our IPL/HA scripts enables us to have everything across all sites stopped, started, or recycled.

By choosing the recyle action every site picks up a current copy of the global site saving us the greif of turning over every stone.

This approach is often used in our test cloverleaf environment but many times in prodcution requires coordinating in conjunction with our quarterly fail-over at 2AM on Sunday.

Russ Ross
RussRoss318@gmail.com

[Jennifer Hardesty]

You keep your SMAT files in a separate area outside of the sites?

Also, out of curiosity, about what size are your SMAT files before they get cycled at the 3 hour limit?

[Russ Ross]

The SMAT files are in the site while accumalaiting for the 3 hour time slice, but are cycled/saved and then archived via tar and gzipped into another filesystem.

As far as how large they are, they come in all different sizes as you might imagine.

Looking for some of the bigger 3 hour SMAT files that are tarred/zipped they are upwards to 50 MBs in Cloverleaf 5.6 and 70 MBs in Cloverleaf 6.0 which produces bigger SMAT files because of extra metadata now being added to the SMAT files.

For a slower interface the SMAT file might be as small as 1 MB.

The file system we use to archive our SMAT in is 500 GBs for Cloverleaf 6.0 with the hopes that will be enough to give us one year of saved SMAT.

In Cloverleaf 5.6 we had 200 GBs of /archived that would only hold 9 months of SMAT.

We are going to probably add EPIC to the mix and that might put us back to less than one year of saved SMAT but time will tell.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Finished a 2AM Sunday golive and while doing some clovertech reading, thought you might be wondering how I came up with our change control approach.

The method of

Code:

not changing current PROD and making a cloned version and changing that, then using a symbolic link pointer to the version you want activated

Was a change management technique I learned from Texaco (now part of Chevron) where I was a Sr Scientific program and was mentored and eventually became one of the 2 Quality Control managers over about 50 programmers, setting best practices and coding standards.

This approach was learned from my years at Texaco with their STARPAK application that has simply been carried forward to my years with the Cloverleaf application.

Another approach I learned at Texaco that I would advise.

Only have 2 people be able to make changes to PROD so they are controlled and have a consistant quality.

The reason for 2 people instead of one is in case one is unavailable and there can be a second set of eyes.

Unfortunately, if you allow any integration person on your team to change PROD then quality will suffer.

At Texaco developers placed their changes in a holding area to be picked up with our montly changes and reviewed with a suite of scripts for quality standards before I would put them in PROD.

I was not allowed to be that strict here and every integration engineer can update PROD, which has caused inconsistant quility standards adherence.

At Texaco, if I kicked back a golive in the holding que because of non-standards compliance that was a quantifiable reflection on the programmers work ethics with not being a team player and counted against them on their annual job review.

This quickly revealed those developers that were habitually slopy and compliance was high because of this approach, whereas without it I have observed standards compliance is much lower and overlooked.

Russ Ross
RussRoss318@gmail.com

[Richard Hart]

We use CVS (migrated from RCS many years ago).

We work on the (Linux) server, so CVS is easier; we have access to all the linux tools as well.

We use scripts for most change control processing.

Our master site provides the

[Author]

Replies