Building My Network

Over the past three years, I’ve spent more time than I care to admit on a single project. It started off innocently. Since then it’s grown to occupy a significant place in my daily thoughts.

As you read this, you’re experiencing the product of these efforts. Perhaps you clicked on an eth.pw short URL to get here. If not, your browser still had to figure out how to get to ethitter.com. Now that you’re here, maybe you need to contact me via email. To make any of these endeavors possible, I’ve built a personal “network” of servers and had endless fun doing so.

A Story

I started off just wanting to host WordPress and have more flexibility in what I did with it. I also wanted a faster experience, one that I couldn’t get on shared/managed hosting. Then came SSL, and myriad performance improvements, followed by a CDN. Eventually, tweaking and futzing with my web presence wasn’t sufficiently engaging.

How else could I complicate life? Running a mailserver? Surely, this is a worthwhile endeavor. But if I do this, I need redundancy–failover for when I bork the primary machine. Thus the empire begins.

With relays in Amsterdam and Singapore (Facebook continues to prefer the latter for some odd reason), I’m eager to try something new. If I can run a mailserver, why not host my own nameservers?

Why not? Because of the considerations and the risk. Glue records, IPv4 and IPv6, recursion, amplification, and more. But still, this all makes it yet more fun. A single server is both the authoritative DNS for my domain, and the primary mailserver. So the foreign relays become nameserver backups as well. If the primary server goes down, I’ll be fine as long as I can bring it back up eventually. Which means backups.

Enter server four, a storage VPS with 700 GB of RAID 50 so that I can keep hourly backups of the other three servers, along with full backups taken every three days. And why not retain six months of those full backups, since the space is available after all?

The Details

My “network” consists of four servers hosted with three different providers. Linode is my primary resource, with Digital Ocean providing failover. An unnamed third host is utilized for backup.

What’s on each?

mail.ethitter.com (ns1.ethitter.com)

• WordPress multisite network (aka what made it possible for you to read this)
  • PHP 7
  • Redis object cache
  • Batcache page cache
  • Concatenation and minification care of Automattic
  • Redis user session storage
  • KeyCDN to front concatenated/minified assets and serve uploaded files (aka my photoblog)
• MySQL 5.6
• nsd nameserver daemon
• Mailserver
  • Postfix MTA with opportunistic TLS
  • Dovecot for IMAP
  • Postfixadmin for creating all of the aliases I want 🙂
  • Amavis for spam and virus screening, plus DKIM and SPF signing
  • Rainloop for webmail
• GitLab CE, a GitHub clone
• Solr under Tomcat
  • Indexes email as well as WordPress content
  • All public connections to Tomcat are made via nginx proxy
• YOURLS link shortener
• Piwik (PHP 5.6) web analytics
• ZNC IRC bouncer
• Prosody XMPP server (alerts and debugging, primarily)
• Nagios monitoring

mail2.ethitter.com (ns2.ethitter.com)

• Postfix
• nsd nameserver daemon

mail3.ethitter.com (ns3.ethitter.com)

• Postfix
• nsd nameserver daemon

Backup Host

• BackupPC
• Let’s Encrypt for SSL (testing only)

Errata

All SSL is provided by StartSSL, though I’ve begun testing Let’s Encrypt in limited situations. I have a Class 2 verification with StartSSL, and Let’s Encrypt doesn’t support wildcard certificates, so I’m unlikely to switch to Let’s Encrypt anytime soon. The short lifetime (three months) of their certificates is also a concern, as they’re used in many places and in various formats.

Someday I’ll write a post about the endless and repetitive monitoring I’ve configured. Most of my monthly traffic is from that and bots, not real visitors.

Conclusion

Given the infrequency of my posts, I’m far more interested in configuring the servers that make this all possible, rather than actually publishing content on this site. Oh well. It keeps me entertained.