Monitoring and culling stale GitLab Runner instances

After I set up GitLab’s continuous integration using DigitalOcean Droplets for autoscaling, I noticed that sometimes, a runner would fail, abandoning a Droplet that I’d manually need to destroy. While this problem was infrequent, it was troublesome-enough to warrant some automated solution. Not finding anything readily available, and thanks to DigitalOcean’s godo library, I put together a Go program to periodically cull stale Droplets.

Continue reading Monitoring and culling stale GitLab Runner instances

Four years of HSTS preloading

I recently wondered how long it had been since I added ethitter.com to the Strict Transport Security preload list, as it’s been several years. I added my domain without fully understanding the consequences, though I’ve been fortunate to avoid any problems that could’ve resulted.

Getting back to my original question, version control made it easy to find August 18, 2014 as the date my domain landed in Chromium’s list: https://src.chromium.org/viewvc/chrome?view=revision&revision=290306. At the time, it was one of less than 1,000 domains in the preload list; the bulk of the list was comprised of Google’s own domains. There are currently over 40,000 preloaded domains in Chrome/Chromium. It’s a good thing preloading hasn’t been an issue, it takes a while to get off the list.

docker-machine doesn’t require Docker locally

I took a break from my ongoing “convert everything to Ansible” project to use the base I’ve established there to install something new: GitLab Runner using Docker Machine. I finally have CI/CD built into my GitLab instance: https://git.ethitter.com/debian/eth-log-alerting/pipelines! 🎉

Continue reading docker-machine doesn’t require Docker locally

Restoring performance after Spectre updates

Two weeks ago, the VPS that hosts this site moved to a machine that had been patched for the Spectre vulnerabilities. Immediately, I began receiving warnings about high load, and these alerts continued unabated for over a week. I tried moving services to other hosts, and I reduced the resources allocated to nginx and php-fpm, all to no avail.

As I continued to monitor and debug the situation, fail2ban regularly appeared among the top resource consumers, but I didn’t think much of it; fail2ban has always been a voracious resource user, but it’s an indispensable tool, so removing it wasn’t an option.

Continue reading Restoring performance after Spectre updates

A Better dyndnsd Client

Previously, I used curl to trigger dyndnsd updates via my Raspberry Pis. This worked well for many months, but lacked IPv6 support as dyndnsd was interpreting my IP from the request. Fortunately, the daemon accepts parameters for IPv4 and IPv6 addresses, so I wrote a Go program to handle regular updates. It still relies on cron, but passes explicit IP values and moves all options to a configuration file.

The client is available from https://git.ethitter.com/open-source/dyndnsd-client. I don't provide built binaries yet, but I'd like to soon.

If your ISP doesn't support IPv6, or if you run multiple daemons on the same network, options are available for your situation. Take a look at the readme for more.

Hopefully someone else finds this useful!

GitLab on the move again

After a few successful months of testing Packet.net, I've once again moved git.ethitter.com. The decision was purely financial–my GitLab instance doesn't receive enough traffic to warrant Packet.net's pricing. As far as reliability and value were concerned, Packet.net was excellent. I would've appreciated built-in backups, but otherwise, I have no complaints about the service.

It will likely come as little surprise that git.ethitter.com is back on Linode. Compared to Digital Ocean, Linode is slightly more-generous with its resources, and GitLab wants all the resources it can get.

The migration itself was quite easy, with most of the time was spent preparing the server; GitLab's backup/restore process did most of the hard work. Now I just have to finish the ancillary setup, like monitoring.

Managing domain and certificate expiration with DomainMOD

With 40 domains–plus a half dozen certificates–to track, I added the DomainMOD tool to my repertoire. Its API integrations, in particular, made it an appealing choice, as I had little desire to manually enter so many details. After three months, I’m quite pleased with my decision.

Installation was as straightforward as a git checkout, creation of a MySQL table, and the addition of a server block to my nginx configuration. With DomainMOD successfully running, I configured it to use my mailserver, then got to importing my domains.

Continue reading Managing domain and certificate expiration with DomainMOD