Four techniques for monitoring server logins

With four Debian servers, each available over both IPv4 and IPv6, login attempts come from many sources. Gabriel Koen, in response to my backups post, asked what I do to monitor logins.

Currently, I use four tools to deal with login monitoring, equally split between proactive and reactive solutions. I don’t believe that any of these are Debian-specific, but I have no relevant experience with other distributions.

My solutions leverage:

• rsyslogd
• Simple Log Watch, better known as swatch or swatchdog
• fail2ban
• logwatch

What follows is how I’ve configured each of these tools.

Caveat

I’ve configured each of my servers to use only key-based authentication. As a result, these solutions are tailored to, or expecting that, password authentication isn’t supported. I’ll post soon about how I made this change, but throughout this post, bear this in mind.

rsyslogd

The first monitor I implemented leverages rsyslogd‘s email capabilities. The configuration I use is a legacy one, I realize, but it works.

The following can be added to /etc/rsyslog.conf, but I prefer to keep it separate and placed it in a .conf file in /etc/rsyslog.d.

$ModLoad ommail
$template mailSubject,"Login Alert: %hostname%"
$template mailBody,"\n\n%msg%"
$ActionMailSMTPServer localhost
$ActionMailFrom alerts@example.com
$ActionMailSubject mailSubject
$ActionExecOnlyOnceEveryInterval 1
# the if ... then ... mailBody must be on one line!
$ActionMailTo alert@example.com
if $msg contains 'Accepted publickey for' then :ommail:;mailBody
if $msg contains 'pam_unix(sshd:session): session opened' then :ommail:;mailBody
if $msg contains 'pam_unix(login:session): session opened' then :ommail:;mailBody
$ActionMailTo alert2@example.com
if $msg contains 'Accepted publickey for' then :ommail:;mailBody
if $msg contains 'pam_unix(sshd:session): session opened' then :ommail:;mailBody
if $msg contains 'pam_unix(login:session): session opened' then :ommail:;mailBody

After making these additions, restart the rsyslog daemon.

Note that I’ve left localhost as the SMTP server because each of my VPS either hosts a Postfix instance that sends mail on behalf of the example.com domain, or the VPS is otherwise configured to route through my primary MX. Posts about that forthcoming.

It’s also noteworthy that the latter two rules are more verbose so that cron-initiated sessions (pam_unix(cron:session)) don’t trigger alerts.

Most credit for the preceding config goes to:

• Creek‘s answer at Monitor all login attempts
• ommail: Mail Output Module

swatch

This Perl-based tool comes from http://sourceforge.net/projects/swatch/.

To start, download the package from the above URL and expand it to a directory in /tmp/. The swatch watcher is compiled and installed, so using /tmp/ is fine.

swatch depends on two Perl modules that you’re likely missing: Date::Calc and File::Tail; I’m zero for four, for what it’s worth. Fortunately, both are available via cpan:

cpan install Date::Calc File::Tail

If you don’t have cpan, Google is your friend. Be sure to cpan upgrade first.

Once those prerequisites are satisfied, installing swatch is a matter of executing these commands in the directory where you extracted the swatch archive:

perl Makefile.PL
make && make test
make install && make realclean

Once the install’s complete, confirm where swatchdog was installed using:

which swatchdog

Note the path output by the last command above as you may need to update it in the CMD portion of the init script. If swatchdog isn’t at /usr/local/bin/swatchdog in your instance, adjust the following accordingly.

Create /etc/init.d/swatch with these contents:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          swatch
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: swatchdog
# Description:       Simple Log Watcher Program
### END INIT INFO
DNAME='swatch'
PIDFILE='/var/run/swatch.pid'
CMD='/usr/local/bin/swatchdog --daemon --config-file=/etc/swatch.conf --tail-file=/var/log/auth.log --pid-file=/var/run/swatch.pid'
case "$1" in
'start'|'restart')
		if [ -f "$PIDFILE" ]; then
			PID=$(head -n 1 $PIDFILE)
			echo "Stopping $DNAME with pid $PID..."
			kill $PID
			rm $PIDFILE
		fi
		echo "Starting $DNAME..."
		$CMD
		;;
'stop')
		if [ -f "$PIDFILE" ]; then
			PID=$(head -n 1 $PIDFILE)
			echo "Stopping $DNAME with pid $PID..."
			kill $PID
			rm $PIDFILE
		else
			echo "$DNAME isn't running."
		fi
		;;
*)
		echo "Usage: $0 { start | stop | restart }"
		;;
esac
exit 0

Ensure that the init script can run using this command:

chmod +x /etc/init.d/swatch

/etc/swatch.conf contains the following in my particular case:

watchfor /Accepted/
	mail addresses=alerts\@example.com, subject=Login Alert
watchfor /ssh:session\): session opened/
	mail addresses=alerts\@example.com, subject=Login Alert
watchfor /login:session\): session opened/
	mail addresses=alerts\@example.com, subject=Login Alert

Lastly, the following ensures the daemon runs when the server starts:

update-rc.d swatch defaults

I gleaned much of this from http://www.linux-mag.com/id/7807/, then turned it around to monitor successful attempts rather than failures. As my caveat notes, I’m only concerned with key-based authentication as that’s all I support, so I can’t guarantee that my configuration will capture all password-based attempts.

fail2ban

Other than the ssh and nginx jails that fail2ban provides, my only significant addition comes from fellow Automattician Konstantin Kovshenin’s post “Fail2ban + WordPress + Nginx.”

logwatch

Little of my logwatch config is original. I started from another of Digital Ocean’s excellent tutorials–How To Install and Use Logwatch Log Analyzer and Reporter on a VPS–which landed my configurations in /usr/share/logwatch/default.conf/. To /usr/share/logwatch/default.conf/logwatch.conf, I made the updates noted below.

I first changed the email addresses used to send and receive notifications:

MailTo = alerts@example.com
MailFrom = logwatch@host

I then revised the log level to provide more detail than most are interested in, and to include any archives that logrotate and its type may generate:

Archives = Yes
Detail = High

Other Notes

My live rsyslogd and swatch configurations send alerts to myriad emails, allowing me to receive those notifications via bridges that deliver to Jabber and Slack. I’m working on ways to remove the email dependence from my notification flow, likely via webhooks.