Introduction
Fail2ban is an important security tool for Linux servers. It monitors authentication and application logs and automatically blocks IP addresses that generate suspicious activity.
For administrators running Apache web servers on AWS Ubuntu EC2, Fail2ban can protect SSH, HTTP and HTTPS services against automated attacks.
However, incorrect detection rules can also block legitimate users, including office networks and trusted public IP addresses.
In this guide, we will cover the essential Fail2ban administration and troubleshooting commands.
1. Check Fail2ban Service Status
Check whether Fail2ban is running:
sudo systemctl status fail2banCheck service status:
sudo systemctl is-active fail2banCheck whether it starts automatically:
sudo systemctl is-enabled fail2ban2. List All Active Jails
sudo fail2ban-client statusA jail is a protection rule that monitors a particular service or log source.
For example:
sshd
apache-auth
apache-badbots
apache-noscript
apache-probes
3. Check Individual Jail Status
sudo fail2ban-client status sshdFor Apache:
sudo fail2ban-client status apache-probesCheck the following information:
Currently failed
Total failed
Currently banned
Total banned
Banned IP list
4. Check Fail2ban Logs
View recent logs:
sudo tail -100 /var/log/fail2ban.logMonitor logs in real time:
sudo tail -f /var/log/fail2ban.logSearch for ban and unban activities:
sudo grep -E 'Ban |Unban |ERROR' /var/log/fail2ban.logSearch for a specific IP:
sudo grep -F '203.0.113.25' /var/log/fail2ban.logYou can also inspect system logs:
sudo journalctl -u fail2ban -n 100 --no-pager5. Check All Banned IP Addresses
Run this command to inspect all active jails:
for jail in $(sudo fail2ban-client status | sed -n 's/.*Jail list:[[:space:]]*//p' | tr ',' ' '); do
echo "========== $jail =========="
sudo fail2ban-client status "$jail"
doneThis is useful when you do not know which jail has blocked a particular IP.
6. Unban an IP Address
If a legitimate office or trusted public IP has been banned, remove it from the relevant jail.
Example:
sudo fail2ban-client set apache-probes unbanip 203.0.113.25For SSH:
sudo fail2ban-client set sshd unbanip 203.0.113.25Important: If the IP is banned in multiple jails, unban it from each relevant jail.
Verify:
sudo fail2ban-client status apache-probes7. Add an IP to the Fail2ban Whitelist
To prevent a trusted IP from being banned again, configure ignoreip.
Open the local configuration:
sudo nano /etc/fail2ban/jail.localUnder the existing [DEFAULT] section:
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 203.0.113.25This applies to all jails unless overridden by a jail-specific configuration.
Alternatively, whitelist an IP in a specific jail:
[apache-probes]
ignoreip = 127.0.0.1/8 ::1 203.0.113.25Add this to the existing jail section. Do not create duplicate section headers in the same configuration file.
Only whitelist trusted, fixed public IP addresses.
8. Reload Fail2ban Configuration
After modifying configuration files, validate the configuration first:
sudo fail2ban-client -tExpected output:
OK: configuration test is successfulReload the configuration:
sudo fail2ban-client reloadVerify the whitelist:
sudo fail2ban-client get apache-probes ignoreipRemember: Reloading configuration does not necessarily remove an existing ban. Unban a trusted IP separately.
9. Check Apache HTTP and HTTPS Ports
Check whether Apache is listening:
sudo ss -lntp | grep -E ':(80|443)\b'Check Apache service:
sudo systemctl status apache2Test HTTP:
curl -I http://127.0.0.1/Test HTTPS:
curl -kI https://127.0.0.1/10. Check Ubuntu Firewall
Check UFW:
sudo ufw status verboseCheck numbered rules:
sudo ufw status numberedCheck iptables:
sudo iptables -L INPUT -n -v --line-numbersCheck nftables:
sudo nft list rulesetFor a publicly accessible website, verify that TCP ports 80 and 443 are allowed.
Also inspect AWS Security Groups and Network ACLs separately.
11. Monitor Incoming Connections with tcpdump
This is particularly useful when a website works from some networks but not others.
Run:
sudo tcpdump -nn -i any 'tcp port 80 or tcp port 443'To monitor one specific client:
sudo tcpdump -nn -i any 'host 203.0.113.25 and (tcp port 80 or tcp port 443)'TCP handshake interpretation:
SYN: Client requests a connection.
SYN-ACK: Server responds.
ACK: Client completes the connection.
Repeated SYN packets without a SYN-ACK indicate that the TCP handshake is not completing. Investigate host firewall rules, Fail2ban actions, routing and other network controls.
12. Common Fail2ban Troubleshooting Commands
Check all important components:
sudo systemctl status fail2ban
sudo fail2ban-client status
sudo tail -100 /var/log/fail2ban.log
sudo ufw status verbose
sudo iptables -L INPUT -n -v --line-numbers
sudo ss -lntp | grep -E ':(80|443)\b'Validate configuration:
sudo fail2ban-client -tReload:
sudo fail2ban-client reload13. Real-World Troubleshooting Case Study
Recently, an AWS Ubuntu EC2 server hosting multiple Apache websites had an unusual connectivity problem.
Problem:
Websites were accessible from other internet connections.
Office users could not access the websites.
Ping and traceroute were successful.
Apache was listening on port 443.
UFW allowed incoming TCP 80 and 443.
During investigation, tcpdump showed repeated SYN packets from the office public IP without a completed TCP handshake.
The decisive discovery came from:
sudo fail2ban-client status apache-probesThe office public IP appeared in the banned IP list.
Root cause: Fail2ban had automatically banned a legitimate office public IP through the apache-probes jail.
Resolution:
sudo fail2ban-client set apache-probes unbanip 203.0.113.25The trusted IP was then added to the existing jail's ignoreip configuration.
After validating and reloading Fail2ban, the administrator could verify the whitelist and retest connectivity.
This incident demonstrates why checking Fail2ban's active banned IP lists is an important troubleshooting step when only certain public IP addresses cannot access a website.
14. Important Security Recommendations
Keep Fail2ban enabled.
Whitelist only trusted public IP addresses.
Regularly review banned IP addresses.
Monitor Apache access and error logs.
Keep SSH access restricted to trusted networks.
Back up Fail2ban configuration before modifying it.
Validate configuration before reloading.
Do not flush firewall rules or disable UFW without a clear recovery plan.
Review AWS Security Groups and Network ACLs independently.
Conclusion
Fail2ban is an effective security solution for protecting Ubuntu servers and Apache websites.
However, incorrect filtering can block legitimate users and business networks.
Understanding jails, logs, banned IPs, whitelist configuration and firewall rules allows administrators to troubleshoot these problems safely without unnecessarily restarting production services.
Useful references:
0 Comments