Fail2ban Behind Reverse Proxy

Assumptions, Prerequisites, Prior Set Up

This guide is for when the reverse proxy & FoundryVTT server are both on separate machines

This guide builds upon the prior one and details how to setup fail2ban to block bad bots, automated login attempts, and repeated connection attempts that are likely malicious, when your server is behind a reverse proxy. 

If you haven't taken a look already my prior post detailed how to go about setting up fail2ban rules.
https://blog.iterlyze.com/2024/11/fail2ban-foundryvtt.html 

This guide assumes you are using a Debian 12 server (or Ubuntu) and have a basic idea of how to use a Linux shell. Some other assumptions that won't affect setting up fail2ban include that you are using Nginx as the reverse proxy, nftables as your system's firewall frontend, and that you are forwarding traffic to a FoundryVTT server.

Make sure the reverse proxy and the server it sends traffic to is working before trying to set up fail2ban. If you are setting up FoundryVTT & a reverse proxy for the first time the official FoundryVTT site has some great step by step guides you can follow. You can use Apache (widely used), Nginx (fast & efficient), or Caddy (easy to use).
Recommended Linux Installation Guide | Foundry VTT Community Wiki

Caddy Proxy Server | Foundry Virtual Tabletop
Nginx Proxy Server | Foundry Virtual Tabletop
Apache Proxy Server | Foundry Virtual Tabletop

If you have trouble with the Linux commands the following blog post may help https://blog.iterlyze.com/2020/10/i-recently-made-quick-draft-of-useful.html
(If you need help with the basics i'm happy to answer questions)

Introduction

These steps should work for any reverse proxy & log file you want fail2ban to take a look at. The issue when using a reverse proxy between your servers and the internet is that fail2ban and your firewall primarily block at the network/transport layer while the ip address of the connecting device is encapsulated by your reverse proxy in the application layer. While there are some features you can use for looking at strings in the application layer, if you are using https you won't be able (unless you use a different setup). The solution is to have fail2ban operate on your reverse proxy instead of the server. I'll include the commands for setting up fail2ban at the end (Starting on step 8) with less explanation than the previous post. If you run into problems, start with the simple setup then slowly add to it.

Login to the root account, with su or use sudo before running the commands below.

Step 1) Install needed software

On the server (FoundryVTT) install the needed software. We will be using the lsyncd which in turn uses rsync to send/update logfile changes to our reverse proxy. lsyncd allows for finding changes in files in seconds from the time the files have been changed.

apt-get update; 
apt-get install -y lsyncd;

On the reverse proxy install fail2ban and a firewall (If you want to use nftables as your firewall).

apt-get update;
apt-get install fail2ban nftables;

Step 2) Create directory on Reverse Proxy

On the reverse proxy create a directory to store your log files

mkdir -p ~/foundryLogs;

Step 3) Create Directory & Configs On Server with Logs (FoundryVTT)

mkdir /var/log/lsyncd; 
touch /var/log/lsyncd/lsyncd.{log,status};
mkdir /etc/lsyncd;

In order for lsyncd to work correctly we need to create a location for it to store it's own log files. (idk why when you install the package it doesn't just make this for you). The first command creates a directory in the /var/log folder. The second command creates two files lsyncd.log and lsyncd.status in the /var/log/lsyncd/ folder. The third command creates a directory in the /etc/ folder where you will place the lsyncd configuration called named lsyncd.conf.lua. This file is written using valid Lua syntax, so if you know that programming language it might make things a little bit easier for you to write without error. (Knowing Lua is not needed, I don't know Lua)

Continue with creating the configuration file. You will need to make sure the directories match the ones you have created, that you use absolute paths when defining them, and that the usernames match for the source (FoundryVTT) and destination (rsync section) You might like to note that you can easily find the IP address of each machine by logging in and running hostname -I 

nano /etc/lsyncd/lsyncd.conf.lua    

settings { 	logfile = "/var/log/lsyncd/lsyncd.log",
		statusFile = "/var/log/lsyncd/lsyncd.status",
		inotifyMode = "Modify" }

sync { default.rsyncssh,
	delay = 1,
	source = "/home/username/foundryuserdata/Logs/",
	host = "192.168.1.xxx",
	targetdir = "/home/username/foundryLogs/",
	exclude= { 'error.*','dianogstics.*','news.*','*.json', 'debug.log.*' },
	rsync = { rsh = "/usr/bin/ssh -l username -i /home/username/.ssh/id_ed25519 -o UserKnownHostsFile=/home/username/.ssh/known_hosts" }
}

The first part of the lsyncd.conf.lua file contains the settings function. You define where you want its log files located. In this case in the folders that we created above. Of particular note we will be setting the inotifyMode to Modify. It appears that foundryVTT doesn't CloseWrite, so we need lsyncd to specifically look for modifications to the file instead. This is how we are detecting near realtime changes to the log.

The second part of the lsyncd.conf.lua file contains the sync function which configures how, where, when, who, and what for sending the log file to the reverse proxy.

(How we send changes)

We want the logfile to be sent securely so we will be using rsyncssh by default. 

(When we send changes)

"delay" is how quickly we want to detect that changes have been made to the file. You can set this slower if you want depending partly on how sensitive your fail2ban setup is. 1 is quick, 5 is standard, 10 is acceptable normally. If you have a low resource system you might set the delay to a higher number if needed. 

(Where we send changes from)

"source" is where lsyncd is getting its log files from. In this case the server (FoundryVTT). 

(Where we send changes to)

"host" is the network address the logfiles are being sent to (Reverse Proxy). 

"targetdir" is directory on the host (Reverse Proxy) that the log files are being sent to. 

(What we don't send)

"exclude" is a list of patterns for files that we DON'T want to send. The various versions of FoundryVTT seem to have various different log names, so you might change this list depending on what version that you are using. What's listed should work for v10 and maybe v12.

(Specific Whom & How we send changes)

"rcync" is a set of options for who & how we are sending the log files. This is for configuration of the Rsync utility. Since we really don't need the root account for accessing the log files we should use the useraccount that we are using for foundry. 

We tell lsyncd to tell rsync where the binary for the secure shell (ssh) is.

-l to set the username we will be login to on the reverse proxy. 

-i is to tell rsync where the key for remote login is for the reverse proxy. (more info when we make it)
-o tells rsync where the list of known hosts is.

Step 4) Create SSH keys on both your server (FoundryVTT) and your reverse proxy.

This is different than configuring certificates and ssl for using https. The technology is similar but in this case it's only for keeping your internal communications secure. Start by running these commands on your server (FoundryVTT). Once again you can use hostname -I to find your ip address quickly.

ssh-keygen -t ed25519

ssh-copy-id username@reverse_proxy_address 

Then restart sshd on the reverse proxy and attempt to login.

ssh username@reverse_proxy_address 

Then run the commands again on your reverse proxy.

ssh-keygen -t ed25519

ssh-copy-id username@FoundryVTT_address 

Then restart sshd on the FoundryVTT server and attempt to login.

ssh username@FoundryVTT_Address 

Step 5) Test that lsyncd is working

On your host server (FoundryVTT)

service lsyncd start
systemctl status lsyncd

Also check that the files have been successfully transferred over.
If they haven't, check your firewall configuration, or check again after the next step.

Step 6) Configure your firewall

This is a basic configuration for the firewall on the reverse proxy, skip this step if you use a different firewall.

nano /etc/nftables.conf 

/usr/sbin/nft list ruleset 

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
     chain input {
          type filter hook input priority 2; policy accept;
          ct state established, related accept
          ct state invalid drop
          iif lo counter accept
          ct state new tcp dport 22 accept
          ct state new tcp dport 80 accept
          ct state new tcp dport 443 accept
          ct state new tcp dport 30000 accept
          drop
     }
     chain forward {
          type filter hook forward priority filter; policy drop;
     }
     chain output {
          type filter hook output priority filter; policy accept;
     }
}

There is a lot going on here. First line is what binary interprets this file. (#! is pronounced Crunch Bang. Fight me if you disagree :p [I accept pound bang I guess too...] ) 

Flush Ruleset

Clears out any lingering firewall rules.

table inet filter {

table inet filter - Create a table. Multiple tables are supported. inet means it should work on both IPv4 and IPv6.

     chain input {

Rules are defined at various places you can hook them to as network traffic is being processed. Input is right before any of our applications see things and is the appropriate place for most general firewall rules. Forward, and output are other places you can hook into.

type filter hook input priority 2; policy accept;

This is the first line of our filters, it defines the type, where it hooks, the priority (e.g. the order it is processed compared to everything else), and the default policy action to take after going through everything in that particular chain.

         ct state established, related accept

Allow network connections from already established connections. (e.g. if you (outgoing) tried to talk to a webpage, it should be allowed to respond).

          ct state invalid drop

If the network traffic is sus and doesn't look like anything that should be coming in, don't let it in.

          iif lo accept

Let network traffic from yourself (localhost) to yourself.

          ct state new tcp dport 22 accept

Allow ssh if it's a new connection. (If it isn't new then the rule above for state will let it in)

          ct state new tcp dport 80 accept

Allow new http connections. (Technically its a new state, not connection, but i'm simplifying here. If it isn't new we got another rule for that above)

          ct state new tcp dport 443 accept

This rule is for https, as should be expected just for initiating https.

          ct state new tcp dport 30000 accept

This is our rule to allow connections to FoundryVTT. If you changed the default port you need to change this rule to match what you changed it to.

          drop

Anything else shouldn't be let in. (You might need to add additional ports, say for example if your internal ports don't match what the reverse proxy is accepting)

Step 7) Restart your firewall

systemctl restart nftables

OR reload the rules from your config

/usr/sbin/nft -f /etc/nftables.conf

As a note any time you restart or reload your firewall you also need to do so with fail2ban. Here is a great blog post about how to do that automagically everytime. Using systemd to bind fail2ban to nftables < System | The Art of Web

You can see your currently loaded firewall rules with the following command.

/usr/sbin/nft list ruleset

Test everything above by trying to browse to your FoundryVTT instance.
Assuming that it works, below is a quick config for setting up fail2ban. (If you haven't already read the other post)

The following steps should be done completely on the Reverse Proxy.

Step 8) Create Local Configs

 touch /etc/fail2ban/jail.local; touch /etc/fail2ban/jail.d/local.conf

Step 9) Set your settings

nano /etc/fail2ban/jail.local

[DEFAULT]
findtime = 2h
maxretry = 5
bantime.increment  = true
bantime.rndtime    = 1024
banaction_allports = nftables-allports
banaction          = nftables-allports

[nginx-botsearch]
maxretry = 2

[foundryvtt]
findtime = 3m
maxretry = 3
bantime  = 60
port     = 80
logpath  = /home/username/foundryuserdata/Logs/debug.log
backend  = polling
ignoreip = 192.168.1.0/24

nano /etc/fail2ban/jail.d/local.conf

[sshd]
enabled = true

[nginx-bad-request]
enabled = true

[nginx-botsearch]
enabled = true

[nginx-http-auth]
enabled = true

[foundryvtt]
enabled = true

echo "sshd_backend = systemd" >> /etc/fail2ban/paths-debian.conf 

Added some settings that aren't included in the other post as a hint to more things you might want to consider using. Assuming Nginx as the reverse proxy.

Step 10) Create a regular expression to detect log entries

nano /etc/fail2ban/filter.d/foundryvtt.conf

[Definition]

failregex = ^\{\"ip\":\"<HOST>\",\"level\":\"warn\",\"message\":\"((Administrator authentication failed for session [a-zA-Z0-9]{24}; invalid password)|(User authentication failed for user [a-zA-Z0-9]{1,50}; invalid password\",\"session\":\"[a-zA-Z0-9]{24}))\",\"status\":40[13],\"timestamp\":\"\"}$

Step 11) Test 

systemctl restart nftables; systemctl restart fail2ban 

Try to get yourself banned, and then issue the command to unban all ip's fail2ban-client unban -all 

If you notice, there are some discrepancies in the settings i've listed, it means you can try your own values and customize it to what appears to work best for your system.

Conclusion

Well there we have it. Hopefully by now you've been able to setup fail2ban when you are using a reverse proxy. At the time of this post (2024) there seem to be some issues with fail2ban and the default sshd jail on debian you can try this sudo echo "sshd_backend = systemd" >> /etc/fail2ban/paths-debian.conf Good luck, if you have problems feel free to leave a comment.