Don’t look now, but people have been trying to break into your server. Don’t believe me? Okay, look now. Type this command (as root) in your shell:
grep "Invalid user" /var/log/secure | wc -l
The number you see is the number of times someone has tried to log into your server via ssh with an invalid username. Some of them might be you, but most of them are not. Most of them are people (or robots) all over the Internet who have tried to log into your server – and they’re not doing it in order to lend a helping hand in running the thing.
It isn’t that there’s anything wrong with your setup, it just that your server is connected to the Internet, and any time you connect anything to the Internet people immediately begin trying to wander in uninvited. Aside from disconnecting it from the Internet altogether (which really sort of hampers its ability to perform in its role as a server), there are a few things you can do to help protect your host.
This tutorial will guide you through the installation of two excellent (and free!) utilities that work in concert to help guard against many kinds of attacks. I assume some familiarity with issuing commands at a Unix shell, and proficiency with a text editor such as vi, pico, nano, or emacs. A working knowledge of IP networking in general will also be beneficial, but isn’t strictly required (at a minimum, though, you need to know which ports you want to allow through your firewall). We’ll be installing the following:
The first line of defense for any host or network is usually a firewall of some sort. APF (Advanced Policy Firewall) is a firewall tool provided by R-fx Networks that layers atop the Linux kernel’s iptables packet filtering utility. By itself, iptables is a great static packet filter, but APF adds a number of dynamic behaviors such as reactive address blocking (which is every bit as cool as it sounds), and the ability to subscribe to some popular lists of networks and addresses which are known for being sources of attacks.
Also from R-fx Networks, Brute Force Detection is a utility that watches system log files for repeated login attempts (like all those failed ssh connections) to various services, and automatically blocks the originating IP address. It works in conjunction with APF (when BFD detects an attacker, it tells APF to block the attacker’s address).
If you’re in a hurry and just want to get a basic, minimal setup, these quick start instructions are enough to provide a reasonable setup that will suit most needs. Beyond these, we’ll dive a little bit deeper into each of these utilities to explore some of the more interesting options, and to learn about customizing their behaviors to your specific needs. It’s well worth the time investment.
Run the following commands (at least the last two must be run with root privileges):
# curl -O "http://rfxnetworks.com/downloads/apf-current.tar.gz" # tar -zxvf apf-<version>.tar.gz # cd apf-<version> # ./install.sh # chkconfig apf on
APF’s main configuration file is at /etc/apf/conf.apf. Open that file in your favorite text editor, and make the following changes:
Find the line that begins with IG_TCP_CPORTS and change it to only list only the ports you intend to leave accessible to the Internet at large. For me this is only 22 (ssh), 80 (http), and 443 (https) since my server is a web server and I need to be able to log into it remotely. Yours may differ. If you’re unsure which ports go with which services, you can usually find this information by looking at the /etc/services file on your host.
Attempts to connect to any TCP ports not listed there will be met with a blank stare.
Do the same thing with IG_UDP_PORTS. If you don’t know that you need to offer services via UDP (DNS is the most common service that uses this protocol), chances are you don’t. I don’t, so mine is blank.
Turn on reactive address blocking, which enables the firewall to watch for attempts to breach it, and react by blocking the IP address of the attacker. This is one of the main reasons we’re using APF to begin with, so it doesn’t make much sense to leave it off.
This next setting is very important. Important enough to warrant a bold font, even. APF includes a “development mode” setting, and out of the box it’s turned on. With it enabled, your firewall will automatically turn itself off 5 minutes after you start it. That’s good, because the point of it is to help you avoid locking yourself out of your host while you’re fine-tuning your firewall rules (also remember that if you do find yourself unable to access your Cloud Server, you can always use the console available via the web interface). But it’s bad if you leave it that way because, well, your firewall will automatically turn itself off 5 minutes after you start it. For now, leave it on:
But once you’re satisfied that your rules are working the way you intended and you can still access your host with the firewall active, turn development mode off.
By default, APF is configured to block traffic coming from reserved or unallocated blocks of IP space – because if anything says it’s part of such a network, it’s obviously spoofing its address and shouldn’t be trusted. The trouble is that APF’s idea of which blocks are reserved is a little out of date. It stores this information in the /etc/apf/internals/reserved.networks file. Replace that file with this one, which is current with IANA’s list of reserved or unallocated space as of this writing.
Finally, start your firewall:
# /sbin/service apf start
And whenever you’ve made changes, reload the ruleset:
# /usr/local/sbin/apf -r
Follow the bouncing ball:
# curl -O "http://www.rfxnetworks.com/downloads/bfd-current.tar.gz" # tar -zxvf bfd-<version>.tar.gz # chmod +x bfd-<version> # cd bfd-<version> # ./install.sh
If you’re wondering why we set the execute bit on a directory in that set of commands, it’s because BFD comes packaged with it turned off, and you wouldn’t be able to cd into the unpacked directory otherwise.
There’s nothing you really need to configure in BFD, so the quick start crowd can stop here. The rest of us will begin our deeper exploration of BFD and APF configuration.
BFD looks for its configuration file at /usr/local/bfd/conf.bfd, and while there’s not a whole lot in there to change, there are a few settings of interest. By default, BFD allows 15 failed attempts for any given service before it places a block. Personally, I find this a bit too permissive, so I turn it down to just 5:
If you want BFD to email you when it’s blocked someone, you need to change two settings:
If you turn that on, make sure your system has the mail command. If it doesn’t, you can install it like this:
# yum install -y mailx
Finally, you may wish to set the number of events that trigger a block to different values for different services. You can do this by enabling and editing the TRIG lines in the files found in /usr/local/bfd/rules. Any TRIG settings you define there will override the one you set in conf.bfd. Doing this is completely optional.
This is where things can get interesting (and hairy – so if you’re not terribly confident in your knowledge of systems and networks, you probably want to shy away from most of this). APF’s configuration file is very well-commented, and it’s well worth it to spend some time reading through it and understanding what this very capable firewall can do. Here I’ll touch on a couple of the more interesting points, but there’s lots in there to study.
There are organizations that publish lists of networks which are commonly a source of undesirable traffic . APF can subscribe to some of these lists to preemptively block access to your host from some of the dark alleyways of the Internet. Bear in mind that by subscribing to such lists you gain the advantage of a sort of collaborative filtering, but you’re also giving up some amount of control over your firewall insofar as you’re accepting a larger community’s ideas of which networks you should block. Whether you choose to enable these is entirely up to you.
SpamHaus’s DROP (Do not Route Or Peer) list is a collection of networks known to be controlled by spammers, because spammers either own the space, or have taken control of many of the hosts within it (referred to as “zombie” hosts):
Project Honey Pot publishes lists of known email address harvesters, spammers, and smtp attackers:
DShield collects and provides lists of the addresses of some really dark corners of the ‘Net, from which attacks have been known to originate:
We already turned this on earlier, but there are some related settings that are worth a closer look. You should take the time to get familiar with each of them in the configuration file. Here I’ll address a couple that are especially interesting. APF can detect port scans (perhaps the most common first task for a would-be attacker is to scan your host to see what’s listening). You should definitely turn this option on, and consider turning it up. I set my security level to 2 (medium):
How long should an address remain blocked if it triggered APF’s automatic blocking? In my opinion, at least long enough for the attacker to get bored and move on to something else. Who knows, if you block them long enough maybe they’ll reflect on who they are and who they want to be, and decide that trying to break into the machines of others just isn’t the kind of life they want to lead. Yeah, probably not – but I block them for a good 12 hours anyways, just in case I’m wrong:
Finally, you can (and probably should) elect to reset the timer on any blocked address if they don’t get the message and continue to try communicating with your host in any capacity. Think of it as extending their time out for getting up out of their chair before you said it was alright:
A popular saying amongst security folks (who are notoriously fun at parties, by the way – you should think about inviting us more often) is that security is not a product, but a process. APF and BFD are excellent tools for defending a host and are important first steps, but they are not magic bullets that will guarantee your server’s safety. You must remain vigilant, and keep your system as up to date as possible. Make a habit of checking logs regularly (these two log to /var/log/apf_log and /var/log/bfd_log), and be wary of anything suspicious.
Remember, just because you’re paranoid doesn’t mean they’re not out to get you.
Article submitted by Bill Kocik (bkocik.net).
© 2011-2013 Rackspace US, Inc.
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License