Full Report
Kent Ickler // How to Configure Distributed Fail2Ban: Actionable Threat Feed Intelligence Fail2Ban is a system that monitors logs and triggers actions based on those logs. While actions can be […] The post How to Configure Distributed Fail2Ban: Actionable Threat Feed Intelligence appeared first on Black Hills Information Security, Inc..
Analysis Summary
# Best Practices: Distributed Threat Intelligence Blocking using Fail2Ban
## Overview
These practices detail a method for distributing IP blacklisting across multiple Fail2Ban nodes by using a centralized, shared log file synchronized over SSHFS. This transforms individual instance monitoring into a collective defense mechanism where a ban triggered on one node propagates immediately to all participating nodes.
## Key Recommendations
### Immediate Actions
1. **Establish Centralized Log Server:** Designate one server (e.g., a DigitalOcean droplet or internal host) to host the shared global ban log file (`newbans.log`).
2. **Create Dedicated Log User:** On the log server, create a dedicated, unprivileged user (e.g., `f2bcourrier`) specifically for log access and ensure it owns the log directory (`/opt/logdistro`).
3. **Generate SSH Key Pair:** As the dedicated user on the log server, generate a strong RSA 4096-bit SSH key pair and configure the public key in `authorized_keys` for remote append access. **Securely copy the private key**—it is essential for the distributed nodes.
4. **Install Prerequisite Software:** On all Fail2Ban nodes intended to participate in distribution, install the `sshfs` package.
### Short-term Improvements (1-3 months)
1. **Configure SSHFS Mount:** On each Fail2Ban node, use the private key and `fstab` configuration to securely mount the remote `newbans.log` file via SSHFS into a local directory (e.g., `/opt/logdistro`).
2. **Deploy Custom Fail2Ban Actions/Filters:** Deploy the provided custom configuration files:
* Place `iptables-multiport.conf` into `/etc/fail2ban/action.d/`.
* Place custom filters (`newbans.conf`, `sshd.conf`) and the primary configuration (`jail.local`) into their respective directories (`/etc/fail2ban/filter.d/` and `/etc/fail2ban/`).
3. **Configure Global Logging Action:** Modify the Fail2Ban action definitions so that successful local bans also execute a command to append an entry to the mounted `newbans.log` via the `logger` utility, formatted explicitly: `logger -s -i "HOST PORT " 2>> /opt/logdistro/logs/newbans.log`.
4. **Address Log Polling Backend:** If the shared log entries are not being detected, investigate and adjust the `backend` variable within the Fail2Ban configuration on the affected nodes to ensure it correctly recognizes changes in the SSHFS mount.
### Long-term Strategy (3+ months)
1. **Implement Distributed Ban Logic:** Configure Fail2Ban on client nodes to monitor the local mount of `newbans.log`. A single entry in this log file should trigger an *immediate, full-port ban* without waiting for standard local retry attempts (e.g., set `maxretry` to 1 for the `newbans` filter).
2. **Log Rotation Strategy:** Implement robust log rotation on the central log server for `newbans.log` to prevent the file from growing excessively, which could strain SSHFS performance due to excessive data transfer during file monitoring.
3. **Broaden Threat Input (TALL Build):** Integrate other security tools or custom scripts to write sanctioned IP blocks directly to the `newbans.log` using the standardized `logger` command format, expanding the input sources beyond only local Fail2Ban detections.
4. **Scalability Review:** Periodically test the system's performance limits (Wide Build) by increasing the number of participating nodes to preempt bottlenecks associated with SSHFS file concurrency and network latency, considering specialized distributed file systems if necessary for very large deployments.
## Implementation Guidance
### For Small Organizations
* **Simplicity Focus:** Use local network hosting for the log server if latency is a concern and immediate external cloud access is not required.
* **Tooling:** Directly utilize the provided GitHub repository materials to minimize manual configuration errors.
### For Medium Organizations
* **Redundancy Check:** While the core method relies on a single log server, consider implementing a read-only mirror of the `newbans.log` on a secondary server for disaster recovery, though maintaining write synchronization across multiple log servers is complex with this method.
* **Resource Monitoring:** Actively monitor the CPU/network usage on the central log server, as it handles SSH connections from every participating node.
### For Large Enterprises
* **Alternative to SSHFS:** Acknowledge that SSHFS might present scalability/performance limitations for hundreds of nodes. Investigate using enterprise-grade, high-throughput, low-latency distributed log management solutions (e.g., Kafka, centralized SIEM ingestion pipeline) as an advanced replacement for the SSHFS mechanism.
* **Policy Granularity:** Develop complex rules within the `newbans.conf` filter to enable advanced logic, such as: "Only global-ban if a source had three invalid SSH attempts on two different servers within 30 minutes."
## Configuration Examples
The core mechanism relies on this command structure to write a remote ban entry:
bash
logger -s -i "HOST PORT " 2>> /opt/logdistro/logs/newbans.log
When a node detects a ban, it uses a custom action to propagate it. A ban detected locally (e.g., 3 SSH retries) causes the node to execute the action which results in a write to the shared log. When other nodes see this entry locally via SSHFS:
* They immediately apply the block using the `iptables-multiport` action.
* Because they use the same propagation action, they will also write their confirmation back to `newbans.log`, resulting in log "noise" but confirming distributed enforcement.
## Compliance Alignment
* **NIST SP 800-53 (AC-6, AU-6):** Enhances Access Control (AC) by reducing the attack surface via shared rapid blocking. Improves Audit Logging (AU) by centralizing and reacting to authentication events.
* **CIS Benchmarks (Control 12: Audit/Logging):** Improves the detection and response time to unauthorized access attempts across the infrastructure perimeter.
* **ISO/IEC 27001 (A.12.4):** Requires the systematic monitoring of systems and timely response to security alerts.
## Common Pitfalls to Avoid
* **Ignoring `backend` Variable:** Assuming standard log polling will work perfectly with an SSHFS mount. If bans from the shared log are missed, check the Fail2Ban `backend` configuration for the affected nodes.
* **Insecure Key Management:** Failing to secure the private key copied from the log server user (`f2bcourrier`). If compromised, an attacker could write malicious IP addresses to `newbans.log`.
* **Single Point of Failure:** Relying solely on the central log server. If it goes offline, no new distributed bans can be registered or propagated, although already banned IPs remain blocked until local unban timers expire.
* **Ignoring Log Noise:** Accepting that the system writes confirmation entries back to the log. Plan to manage or filter this noise, especially in high-traffic environments.
## Resources
* **SSHFS Documentation:** [https://github.com/libfuse/sshfs](https://github.com/libfuse/sshfs)
* **Fail2Ban Official Site:** [https://www.fail2ban.org/](https://www.fail2ban.org/)