Tracking discrete server configurations

As I’ve written about previously, I currently maintain four servers that together support the many pieces of my online presence. There’s relatively little configuration overlap between each instance, so I wasn’t particularly interested in a solution like Capistrano, Chef, or Puppet. My need was for tracking iterations, not repeating setups, so I leveraged Git.

How?

In the root of each server, I initialize a Git repository and create a README.md. The readme contains pertinent details like the server’s IPs, provider, and hardware specifications. I also note the services running thereon and any TCP ports they’re bound to.

When setting up a new server, creating the repository is one of the first steps I take. I do so as early as the server can support git, and certainly before installing anything relevant to the server’s intended purpose. With the repo in place, I’m ready to capture all configuration changes.

As I install new software, my first step is always to capture the default configuration and commit that before I make any changes to those files. Doing so ensures I can revert my mistakes when issues arise. My revision history is littered with “initial config” entries like these:

Backing up this history

With years of history tracked in these repositories, their permanence is critical to recovering from a server failure. The repository for my primary host, the machine from which you’re likely reading this post, contains 895 commits, those only dating to July 2014 when I rebuilt the server as a 64-bit host. Its 32-bit predecessor accumulated 584 updates stretching back to January 2013 when this idea struck me.

Initially, I opted for private GitHub repositories as remotes, but then I worried about the cost of this approach as well as my reliance on a third party. I was particularly uneasy with GitHub being the sole backup for such critical data, considering that either loss of control of my account, or a failure of their systems, could potentially lose those years of history. I eventually accepted the cost, but couldn’t ignore the lack of redundancy.

To counter this concern, I added remotes with Bitbucket, providing a second, independent copy of my most-important configurations. Their free plan includes unlimited private repositories for up to five users, which amply satisfied my needs.

I complement these services with a GitLab installation hosted on my primary server, which itself deserves its own post. It benefits from the backup solutions I’ve implemented, ensures that I’m not entirely dependent upon third-party services, and gives me a private place to open issues and contemplate undisclosed vulnerabilities or deficiencies.