Categories
Apache Filesystem Linux Ubuntu

fail2ban in depth

Skipping through my log files I found some concerning entries that were not covered by the existing fail2ban config. While trying to figure the correct regular expressions (regexp) to match these entries I had to take a close look at the things fail2ban provides to make things easier.

Basic configuration

As usual my test are done on latest Ubuntu LTS (24.04), so the fail2ban packages comes pre-configured. I’ll only describe the changes done from here.

Status

So with the shipped configuration only on jail (sshd) is activated:

linux # fail2ban-client status
Status
|- Number of jail:      1
`- Jail list:   sshd

Jails

The list of activated jails is defined here:

linux # cat /etc/fail2ban/jail.d/defaults-debian.conf 
[DEFAULT]
banaction = nftables
banaction_allports = nftables[type=allports]
backend = systemd

[sshd]
enabled = true

There are different ways to override settings in .conf files: You can either add a file and replace its ending with .local or you can use the .d/ directories and create config files there.

So as a first step I want to get mail notifications about banned hosts and I’ll modify the jail config for that (for more details see Actions below):

linux # vi /etc/fail2ban/jail.local
[DEFAULT]
# Send info mails
action = %(action_mwl)s

destemail = root@mydomain.de
sender = fail2ban@mydomain.de

To enable more jails we’ll create a separate config file (to prevent conflicts with the config shipped by the system package):

linux # vi /etc/fail2ban/jail.d/00-local.conf
[apache-excessive]
enabled=true

Filters

As the above is not default in the shipped fail2ban config, we’ll also need to define a matching filter:

linux # vi /etc/fail2ban/filter.d/apache-excessive.conf
[INCLUDES]
before = apache-common.conf
[Definition]
excess_code = 429
failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*" %(excess_code)s
ignoreregex =
datepattern = ^[^\[]*\[({DATE})
              {^LN-BEG}

If you created a new jail/filter, you can test it like this:

linux # fail2ban-regex /var/log/apache2/access.log apache-excessive --print-all-matched --print-no-missed --print-no-ignored
<... all matching lines will be listed here ...>

Filter Backend

While experimenting with my apache logs I finally manged to get a working regexp that succeeded in the above tests. But while running fail2ban itself I could not observe any action, even if the log lines matched.

As it turns out it was using journald only the whole time, so all entries going to extra log files were ignored.

So make sure that your jails really use the log files. Here’s how things looked before the changes:

linux # fail2ban-client status apache-excessive
Status for the jail: apache-excessive
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- Journal matches:  _SYSTEMD_UNIT=apache2.service
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

So our config requires some extra options to use log files instead of journald:

linux # vi /etc/fail2ban/jail.d/00-local.conf
[apache-excessive]
enabled=true
backend = pyinotify
# alternative: backend = polling (better logs in debug mode, but potentially more resource intensive)
port     = http,https
logpath  = %(apache_access_log)s
journalmatch =

After applying these changes and restarting fail2ban the output now lists the analyzed log files correctly:

linux # fail2ban-client status apache-excessive
Status for the jail: apache-excessive
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/apache2/access.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Actions

And finally we need to tell fail2ban what to do in case one of the rules gets triggered.

A typical reaction is to create a blocking firewall rule. In former times iptables was used now looking for any tables related to fail2ban failed.

Turns out the firewall action changed to nftables in the meantime:

linux # nft list tables
table ip filter
table ip6 filter
table ip nat
table inet f2b-table
linux # nft list table inet f2b-table
table inet f2b-table {
        set addr-set-sshd {
                type ipv4_addr
                elements = { 12.34.56.78, 78.56.34.12,
<...>
                             99.78.67.34, 34.56.78.90 }
        }

        chain f2b-chain {
                type filter hook input priority filter - 1; policy accept;
                tcp dport 22 ip saddr @addr-set-sshd reject with icmp port-unreachable
        }
}

While the default action is to just ban/unban IPs we changed that default action in the very beginning.

As it turns out with these settings fail2ban will also send a mail for every single jail started/stopped upon restart. To get rid of that we’ll override some sendmail action settings:

linux # vi /etc/fail2ban/action.d/sendmail-common.local
# Override defaults (sendmail-common.conf)
[Definition]
# Disable email notifications of jails stopping or starting
actionstart =
actionstop =

Useful commands

Test current fail2ban server configuration:

linux # fail2ban-server --test
OK: configuration test is successful

List banned IPs (optionally including their bantime):

linux # fail2ban-client get sshd banip --with-time
12.122.52.119   2025-07-19 12:39:42 + 86400 = 2025-07-20 12:39:42
141.22.44.125  2025-07-19 14:27:09 + 86400 = 2025-07-20 14:27:09
13.15.145.223  2025-07-19 14:27:46 + 86400 = 2025-07-20 14:27:46

And in case you just want to stop one annoying IP address:

linux # fail2ban-client set sshd banip 12.34.56.78
1

Problems encountered

While trying to start one of my fail2ban instances the startup failed. The error seems to be related to the number of opened files:

linux # sysctl fs.inotify.max_user_instances fs.file-max user.max_inotify_instances
fs.inotify.max_user_instances = 128
fs.file-max = 9223372036854775807
user.max_inotify_instances = 128
linux # ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 511592
max locked memory           (kbytes, -l) 16389744
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 511592
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

Fix: Increase number of open files in systemd unit:

linux # cp /lib/systemd/system/fail2ban.service /etc/systemd/system/fail2ban.service
linux # vi /etc/systemd/system/fail2ban.service
<...>
[Service]
# Add this line (s. https://sourceforge.net/p/fail2ban/mailman/message/35884163/)
LimitNOFILE=81920
<...>
Type=simple
<...>
linux # systemctl deamon-reload
linux # systemctl restart fail2ban

Next problem: When calling "fail2ban-client status" fail2ban dies.

linux # fail2ban-client status
<...> fail2ban                [3294423]: ERROR   Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?
linux # tail -f /var/log/fail2ban.log
<...>
fail2ban.asyncserver    [3233358]: ERROR   filedescriptor out of range in select()
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/fail2ban/server/asyncserver.py", line 164, in loop
    poll(timeout)
  File "/usr/lib/python3/dist-packages/asyncore/asyncore.py", line 144, in poll
    r, w, e = select.select(r, w, e, timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: filedescriptor out of range in select()
fail2ban.asyncserver    [3233358]: ERROR   Too many errors - stop logging connection errors
fail2ban.asyncserver    [3233358]: CRITICAL Too many errors - critical count reached {'accept': 0, 'listen': 1001}
fail2ban.server         [3233358]: INFO    Shutdown in progress...
<...>

For now there’s only the workaround mentioned in this issue (change default use_poll from False to True) and restart fail2ban:

linux # vi /usr/lib/python3/dist-packages/fail2ban/server/asyncserver.py
<...>
#    def start(self, sock, force, timeout=None, use_poll=False):
    def start(self, sock, force, timeout=None, use_poll=False):
        self.__worker = threading.current_thread()
<...>
linux # systemctl restart fail2ban

Things learned on the way

I also learned how to get the process id of a systemd service (useful for automating things):

linux # systemctl show --property MainPID --value fail2ban
12345

I also had some trouble while sending notification mails, some of them were sent others failed to be delivered. Turned out to be an ugly DNS resolver inconsistency. You can read more about that here.

Leave a Reply

Your email address will not be published. Required fields are marked *