Distributed fail2ban

Once you start to use fail2ban on more than one machine you’ll get to the point where you’d like to apply the IP blocking to machines other than the local one.

While there are other (partially commercial) solutions to do that, I still decided to go with fail2ban.

Please consider the following description as a proof of concept, that really needs some polishing for production use. It’s only meant to show what can be achieved with fail2ban.

Idea

So what’s the idea (and where did it come from):

I’m using a public server (mostly for its public IP address), but most of the (more resource hungry) applications run on a machine at home (connected via wireguard tunnel).

The forwarding of connections is done via haproxy on the public server. haproxy is forwarding the connections as TCP connections (using the send-proxy extension to transmit the original source IP) – on the receiving side tools capable of that protocol (like apache wiht mod_remoteip) can receive the proxied data from the public server and will use the original IP address (transmitted in the proxy extension) for logging and other purposes.

So while the network connection is coming from the public server, logs will contain the original IP. A default fail2ban installation will parse those logs and block the original IP on the home server without the desired effect: as all connections are proxied by the public server the source IP will always be the IP of that server. The home server will never see a connection from the original source IP and therefore cannot block it.

To make it more specific, let’s assume we have

• “P”: IP of public server (public.mydomain.de)
• “H”: IP of home server (home.mydomain.de)
• “A”: IP of a malicious attacker using

So IP packets from “A” are sent to “P” and from there proxied to “H”. On “H” fail2ban detects malicious activity and bans “A” on “H”. But “H” will never directly receive an IP packet from “A” and therefore local firewall will never be able to block the system “A”.

The missing part therefore is to apply the blocking on the remote server “P” instead of the local “H”.

Realization

On the public server

First we create a new filter and jail (that should never be triggered locally):

linux # vi /etc/fail2ban/filter.d/remote-fail2ban.conf 
[Definition]
failregex = ^\[THIS WILL NEVER EVER HAPPEN - JUST A REMOTE JAIL WITHOUT LOCAL ACTION <HOST> \]
ignoreregex =

An now activate a new jail matching that filter:

linux # vi /etc/fail2ban/jail.d/01-remote-fail2ban.conf 
[remote-fail2ban]
enabled = true
logpath =

Restart fail2ban and were (almost) done on this machine:

linux # systemctl restart fail2ban

On the home server

As we’re planning to trigger the ban action on the public server using ssh we first need to create and distribute ssh keys to secure the connection from home server to public server:

linux # ssh-keygen -t ed25519 -f ~/.ssh/fail2bankey -N '' -C "fail2ban: $(hostname)"
Generating public/private ed25519 key pair.
Your identification has been saved in ~/.ssh/fail2bankey
Your public key has been saved in ~/.ssh/fail2bankey.pub
The key fingerprint is:
SHA256:<...> fail2ban: home.mydomain.de
The key's randomart image is:
+--[ED25519 256]--+
<...>
+----[SHA256]-----+
linux # ssh-copy-id -i ~/.ssh/fail2bankey root@public.mydomain.de
root@public.mydomain.de's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@public.mydomain.de'"
and check to make sure that only the key(s) you wanted were added.

Then we’ll need to define new ban actions including the ssh commands using the generated keys:

linux # vi /etc/fail2ban/action.d/remote-fail2ban-ssh.conf 
[Definition]
actionstart = 
actionstop = 
actioncheck = 
actionban = ssh -i /root/.ssh/fail2bankey root@public.mydomain.de /usr/bin/fail2ban-client set remote-fail2ban banip "<ip>"
actionunban = ssh -i /root/.ssh/fail2bankey root@public.mydomain.de /usr/bin/fail2ban-client set remote-fail2ban unbanip "<ip>"

[Init]
name = default

And we need to make those actions the default ones:

linux # vi /etc/fail2ban/jail.local
[DEFAULT]
# Send info mails
action = %(action_mwl)s

destemail = root@mydomain.de
sender = fail2ban@mydomain.de

banaction = remote-fail2ban-ssh
banaction_allports = remote-fail2ban-ssh

Restart fail2ban on this server too…

linux # systemctl restart fail2ban

…and wait for the first IP ban.

If everything worked as expected you might see something like this on your public host:

linux # fail2ban-client get remote-fail2ban banip
12.34.56.78

And now the IP packets coming in via public host will be blocked 🙂

Drawbacks

As mentioned before one of the drawbacks of the above solution is the ability to execute arbitrary code as root on the public server if you get access to the unprotected ssh key.

So make sure to keep the key as secure as possible. Further steps may include using an unprivileged ssh user in combination with sudo to restrict executable commands or to use the "command=" option in ssh’s ~/.ssh/authorized_keys for that purpose.

Another disadvantage is the difference in bantime between the local fail2ban config and the one on the public server’s remote-fail2ban jail:

• all services blocked remotely will use the same bantime (the local config is completely irrelevant)
• the bantime on the home server may vary from the one on the public server (with may result in unexpected behaviour like repeated re-bans – even if the original ban is still active)