Migrating Cloverleaf changes to production

[Ellen Shenk]

I am interested in hearing about your best practice ideas for moving Cloverleaf changes from test to production.   As you know this could include changes to Xlates, tcl procs, tables, HL7 variants, netconfig, alerts, etc.  

Do you use checklists, templates, or peer review?   Do you have any automated tools to make sure you don

Topic

[Vaughn Skinner]

We have used two main concepts:  revision control and automated testing.

Both production and test environment are in svn.  Before any release, we make sure our test environment passes a set of automated tests.  We migrate changes by updating svn on the production.

This has worked very well for us.  However, we currently have only one developer.  I expect when we need more developers, we will have a central testing environment which we use to confirm that all the tests pass, then svn tag that release and deploy it.

Svn allows us to confirm that the development and production environments are exactly the same.  It also provides us with a way to view and double check the changes being applied.

The automated testing allows us to make changes and be pretty confident we didn’t break any previously working code.  This depends upon the creation of adequate tests.

[Karl Garen]

I would be interested in hearing a little more on how you set up svn.

Karl Garen
Sr. Programmer Analyst
University of Vermont Medical Center
Burlington, Vermont

[Stephan Rubin]

Vaughn,

I too am interested in hearing about your SVN implementation, in particular because you mentioned that you have a single collaborator presently; I am wondering how you see the version and change control workflows you currently have in place scaling as your team grows? Change control becomes exponentially more important as the number of collaborators on a project increases.

Also, you made mention of automated testing you conduct… what sort of tools or framework are you using to carry this out?

[Russ Ross]

Our test sites and prod sites have different architecture so promoting from test to prod means a manual checklist and some testing to be sure it is right.

Of course how do you test in live PORD before you go live to make sure you didn’t miss anything and how do you backout with 100% confidence if things don’t go well?

I will attempt to tell you how I answered these questions by describing what I came up with at our institution.

All our cloverleaf sites are symbolic links to site versions so for example lets take prod_global_adt if we look in $HCIROOT I see this

lrwxrwxrwx [code]lrwxrwxrwx

Russ Ross
RussRoss318@gmail.com

[Vaughn Skinner]

We have only the site directory in svn.  We exclude some directories used by cloverleaf: exec, lock, revisions.

We use a stock svn installation on the test server.  The production server uses ssh to get into it.  We use the command line svn tools.  ‘svn status -u’, ‘svn diff’, ‘svn update’ on both servers.

NetConfig doesn’t seem to stay in sink.  ie when cloverleaf saves it, it sometimes reorders the parts.  I wrote a script which parsed it into sections and then rewrote it ordering it by name of section.  I run that script before checking in NetConfig and then I can use the svn diff tools to tell what changes have occurred.

We use subversion for everything except the live process files.  So tclprocs, xlates, netconfigs, formats, etc.  Also other scripts and sql reports and their output for reference.

We currently only have one site.  But you could create another svn location for another site.  Or use svn externals if you wanted to share tclprocs between sites for instance.

Regarding change control, I envision a development environment for each developer and then a central environment where all changes are merged and tested before tagging a release version.  I don’t know very much about that as I haven’t worked in enviroments with multiple developers in revision control.  I count on unit testing to confirm that all the parts are working together.

Obviously, svn gives you an easy way to revert changes if there is a problem.  I’ve needed that a few times. 🙂

Vaughn

[Vaughn Skinner]

Stephan Rubin wrote:

Vaughn,

Also, you made mention of automated testing you conduct… what sort of tools or framework are you using to carry this out?

We use a homegrown system.  It has a central testing framework which is actually a shell script.  It has a set of functions for different types of tests.  Then a section invoking the various test groups.  Each test group has an IN directory with a set of files each with raw input, and then an OUT directory where the output is saved.  This is compared with a MASTER directory to determine if that individual test passed.

For instance: we test a tclproc with a wrapper around hcitpstest which passes hcitpstest the input file (hl7) and then saves the output file.  For each test condition we create a different input file with a filename that describes the test.

We have additional tests for individual route testing (hciroutetest), series of routes, xlate tests (hcixlttest), and standard output.

We also use the respective language unit test frameworks of tcl for testing tcl libraries and perls for perl libraries.  Many of the custom reports accessing our LIS system are also unit tested in this same process.

The whole set of tests is run every night and I get an email which contains any failed tests.

I have a script which I use to run the emacs ediff command to compare the output with the master files.  This lets me see exactly what lines/letters have changed very quickly.  I can evaluate 10 changed hl7 tests in 2-3 minutes, see all the changes and save the changes I want to the MASTER files.

It could use some tuning and cleanup as it has grown with our need, but it works for us.  It has made it possible for us to make changes, test and put into production within an hour, while being confident that everything else still works.

[Author]

Replies