http://www.linuxpromagazine.com/Online/Features/fail2ban-Intrusion-Prevention-Framework
For its size, fail2ban, a utility that scans logfiles and bans suspicious IP addresses, punches well above its weight.
I dare say that only a few sys admins haven’t heard of
fail2ban – maybe those starting out or those who have focused on different areas. In my experience, it’s quite rare that really small utilities can affect the way you run your servers to the extent that fail2ban has. It certainly explains its popularity.
fail2ban is a feather-weight set of scripts that can easily integrate with popular firewalls and, amongst many other things, catch any failed logins for services that you’re running and then ban the IP address after a certain number of failed attempts. Admittedly that sounds like quite simple functionality, but when you get down to the innards of the software, it’s a truly powerful tool.
I had been using fail2ban on SSH login failures, probably it’s most common usage, before I became increasingly annoyed with web server logs filling up with nefarious probes attempting to compromise PHP with remote exploits (and a myriad of other HTTP attacks). It got to the point at which a large proportion of the Apache logs were failed attempts to find hidden directories or non-existent Joomla installations among the legitimate hits on the websites.
I also ran a few mail servers that allowed mail relaying via
SASL password authentication, which (and there are other ways of running the authentication side) had system user accounts with
PAM checking for correct passwords. I had set the SASL user accounts so that a shell login couldn’t be used to access the server, but I was still more than aware than having a piece of software so readily open to abuse by brute force was far from ideal. So, fail2ban
stepped forward yet again; I could simply ban any IP that entered the wrong password three times for as long as I wanted.
From the scenarios above, I hope you will agree that fail2ban can be applied in all sorts of ways. To give you a head start in this article, I’ll offer some examples, ranging from those straight out of the documentation to those that were hard won. (Those of you who speak
regular expressions, or
regex, as your second language would have found them easy, I’m sure, but I prefer a cogent language that doesn’t involve an aching head coupled with eye strain!)
It Must Be MagicRather than the smoke and mirrors that some pieces of software employ to keep their workings secret, fail2ban is transparent in the work it does behind the scenes.
In simple terms
fail2ban keeps a close eye on your logfiles, and when a specific pattern is matched within those logs, it triggers an action that you’ve predefined within its “action” config files. The distinction between its filters and actions in the directory structure is clear once you’ve installed it.
Script Kiddies and Port 22As I mentioned, the most popular usage of fail2ban is stopping probes that try every username ever conceived by mankind on your poor SSH server on port 22. Not only is it a logging irritation, but it is highly insecure, allowing people limitless tries at guessing your username and password combinations. Now is not the time to detail how to secure your SSH server, but for goodness sake, move it to a port other than port 22, and unless you’ve got a really, really good reason, then permit access by IP address with
TCP Wrappers. Either way, you should consider banning failed logins for a short time in case a poor configuration mistake gives miscreants access inadvertently.
By my reckoning, for user-accessible services on a server, any user who gets a password wrong more than three times is going to need a new password (from a Support Department most likely), so I’d prefer not to allow nefarious types to keep trying repeatedly until they give up. Bear in mind that some scripts attack a port several times a second, so in a minute, they could have just about exhausted all popular logins beginning with the first letter of the alphabet!
Figure 1: An example from the fail2ban site of how a standard SSH logfile might look and what to look for when it reports failed login attempts.
Figure 1 shows an SSH logfile example of failed logins. All you have to do (and please bear in mind my comments about eye-strain and regex earlier) is create a regular expression to catch those failed login attempts so you can trigger a response when they’re spotted in the logs.
Thankfully, a few useful examples are already available post-installation, with a couple of handy tags to save you from figuring out other ways of spotting patterns – one such tag being
. If you’re sitting comfortably, the pattern below matches an SSH login attempt failure which, on a Debian box, lives in the
/etc/fail2ban/filter.d/sshd.conf file. Don’t worry this first example isn’t too trying. The two salient lines in the config file are as follows:
failregex = ^%(__prefix_line)sFailed (?:password|publickey) for .* from (?: port \d*)?(?: ssh\d*)?$
ignoreregex =
By comparing that regex with the logfile in Figure 1, I hope you get the gist of how it pieces together. As I said, don’t be too perturbed by the regex complexity; thankfully, several solid examples are included at installation time.
Fortunately, fail2ban is flexible – not just in the services it can monitor but also across different operating systems, all of which have many versions themselves. So, understanding your potential regex pain, the developers, even in the standard SSH config file, have included a number of
failregex examples, which should match your system’s logfile format. The examples shown below are originally commented out in the config file, ready for you to choose one entry to copy and paste in place of the uncommented line:
failregex = ^%(__prefix_line)s(?:error: PAM: )?Authentication failure for .* from \s*$
^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from\s*$
^%(__prefix_line)sFailed (?:password|publickey) for .* from (?: port \d*)?(?: ssh\d*)?$
^%(__prefix_line)sROOT LOGIN REFUSED.* FROM \s*$
^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from \s*$
^%(__prefix_line)sUser .+ from not allowed because not listed in AllowUsers$
^%(__prefix_line)sauthentication failure; logname=\S* uid=\S* euid=\S* tty=\S* ruser=\S* rhost=(?:\s+user=.*)?\s*$
^%(__prefix_line)srefused connect from \S+ \(\)\s*$
^%(__prefix_line)sAddress .* POSSIBLE BREAK-IN ATTEMPT!*\s*$
^%(__prefix_line)sUser .+ from not allowed because none of user’s groups are listed in AllowGroups\s*$
I still need to explain the
ignoreregex line in the first example. I hope it’s clear, though, what the setting in that line, again written in hieroglyphics (sorry regex), would produce: an IP address or hostname match that should NOT trigger an action if it’s spotted within that
failregex pattern in the logfiles. I’ll give you some
ignoreregex examples a little later.
SASLI briefly touched on SASL and mail server password authentication earlier, as well as my concerns about opening up a username and password combination from
/etc/shadow to the Internet as a serious hole for abuse. To allay my fears, a simple fail2ban instance is now configured to catch bad logins in a highly efficient manner.
Here is the
failregex line for SASL login attempts from the config file
/etc/fail2ban/filter.d/sasl.conf:
failregex = (?i): warning: [-._\w]+\[\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed: authentication failure
A quick hunt through my
/var/log/mail.log files reveals:
Dec 20 22:59:16 Ganymede postfix/smtpd[26681]: warning: a-not-so-innocent.ip-has-been-disguised.com[20.666.92.40]: SASL LOGIN authentication failed: authentication failure
If you squint long enough at the
failregex and the mail server log format (Postfix, in this case), you can see that they should probably tie in with each other. After some testing, you should be confident that they do.
Lay of the LandNow I’ll look a bit closer at how the config files are laid out in fail2ban’s directory structure. The above configuration files live in the
filter.d/ directory. A quick
ls shows a lot of out-of-the-box fail2ban filter examples. A quick warning: if they don’t fit your operating system perfectly, you might be using the search engines for a while, unless you have a diploma in regex. For many scenarios, though, the main site
http://www.fail2ban.org has how-tos, as well as the bundled examples:
apache-404.conf apache-nohome.conf courierlogin.conf pam-generic.conf pure-ftpd.conf sieve.conf vsftpd.conf
apache-auth.conf apache-noscript.conf couriersmtp.conf gssftpd.conf php-url-fopen.conf qmail.conf sshd.conf webmin-auth.conf
apache-badbots.conf apache-overflows.conf cyrus-imap.conf lighttpd-fastcgi.conf postfix.conf sasl.conf sshd.conf.dpkg-dist wuftpd.conf
apache-misc.conf common.conf exim.conf named-refused.conf proftpd.conf sasl.conf.dpkg-dist sshd-ddos.conf xinetd-fail.conf
As you can see, a multitude of examples are ready to go, including a few Apache config files.
The main configuration file is called
jail.conf, and it’s commented with lots of helpful instructions. The
ignoreip setting can take CIDR IP address formats or just single IP addresses, so the Admin doesn’t get locked out accidentally; also, that’s really handy for testing. Additionally, a default setting that applies to all jails (a brief set of rules about a particular service) that don’t have the
ignoreip setting explicitly configured is:
bantime = 1
maxretry = 3
Here, it’s referring to how many opportunities a visitor has to trigger the filter; in this case, it’s set really tight and is probably impractical for most services for which users occasionally make mistakes.
The next setting is which back-end daemon to use for polling the logfiles. In Debian’s case, I set it to
gamin, which is great at checking files like mailboxes and logfiles really frequently without putting system load up.
Finally, the
actions.d directory contains what actions to take
and with which firewall. In the
jail.conf file, other than the level of action required, to all intents and purposes, the remainder of the file is filled with a short config for each service (a jail), switched on or off with
true or
false. Here’s the entry for SASL:
[sasl]
enabled = true
port = smtp
filter = sasl
logpath = /var/log/mail.log
Aside from inheriting the
bantime and
maxretry default setting from above, all you have to think about now is the action setting in
jail.conf, which is relatively straightforward. Also, you can calm fail2ban’s reaction to a trigger; however, I prefer all guns blazing, with all the offending log entries and a whois lookup of the banned IP address mailed to me:
# ban the IP & send an e-mail with whois report and include relevant log lines to the destemail
action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s]
%(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s]
Lights, Camera, ActionNow that we know what to send and when to send it, we simply need to know how to ban the offending IP address. I’ll use the most popular iptables, the kernel’s Netfilter firewall. The file
/etc/fail2ban/action.d/iptables-multiport.conf has these key config lines present:
Actionban: actionban = iptables -I fail2ban- 1 -s -j DROP
Actionunban: actionunban = iptables -D fail2ban- -s -j DROP
To anyone familiar with iptables, the commonplace switches
-I for
INSERT and
-D for
DELETE for the ban and unban settings should be pretty clear. These are editable, as you’d expect if you want to log further to manipulate iptables in some other way.
ApacheSeveral configurations are already in place for Apache, in-hand with the mail POP3/IMAP and FTP services, but I needed a mishmash of the examples, and admittedly it took me ages to get the regex correct.
I had a very specific requirement that I knew no pages were missing on the websites on the web server (which might generate HTTP status response 404s), but I also needed to ignore explicitly some files and directories.
Any other traffic to the web server that I did not specifically allow could therefore be banned if it attempted to access files that didn’t exist or probe other parts of the server looking for PHP exploits, for example. Inside my
filter.d/ directory, I created a filter called
apache-misc. I’ve shortened the example a little so it’s readable, but the first line of the
failregex after the
[A-Z] part effectively drops any request to Apache that doesn’t include a forward slash. You might be surprised that pretty much everything legitimate should.
After the slash and inside the brackets are entries for what requests I want banned. The next line down (which is still part of the same
failregex) beginning with
looks for matches, including the ever-common 404 errors. Finally the
ignoreregex works beautifully, giving some leeway when it’s needed.
[Definition
failregex = .*"[A-Z]* /(cms|user|muieblackcat|db|cpcommerce|wp-login|joomla|awstatstotals|wp-content|wp-includes|pma|phpmyadmin|myadmin|mysql|mysqladmin|sqladmin|mypma|admin|xampp|mysqldb|pmadb|phpmyadmin1|phpmyadmin2).*"
.*\" (502|500|417|416|415|414|413|412|404|405|403|401|400)
ignoreregex = .*\"GET \/(press|mailto|domestic|word).*
The result of applying this filter to Apache is, whereas formerly I might have had a hundred attempts per IP probing the server, now I only have three entries in the logs (which all happened in the first second before
gamin read the logfile and added the iptables rule). I’m delighted with the lack of unwanted logfile noise, and my web servers are safer! If the config is too strict for you, try and remove the 404 entry from the second
failregex line and up the
bantime and
maxretry numbers to see how you get on.
I’ve barely scratched the surface of this topic here, but I hope it will inspire you to look further into how fail2ban can help secure your servers. After you’ve installed it, have a look at how many IP addresses are “already banned” in the fail2ban logfile,
/var/log/fail2ban.log, and see how many attempts you’ve successfully blocked in a day. With its level of reporting and the inherently efficient and controllable configuration, to my mind, it’s a knockout piece of software that should help out any sys admin greatly.
