Full Report
Intro For the longest time I had the idea to implement a notification system that would alert me if someone ever logged in (or tried to login) to an SSH server or XSession on a machine I controlled, using known compromised credentials that were obtained via a data breach or a canary password. In this post I am going to show you how I implemented just that using Canary Tokens.
Analysis Summary
# Best Practices: Covert Credential Compromise Notification
## Overview
These practices focus on implementing a system to proactively alert administrators immediately upon a login attempt (or successful login) utilizing known compromised credentials or specific "canary" credentials on critical servers (specifically SSH, but applicable to other authentication services). The objective is to achieve timely detection using minimal infrastructure overhead.
## Key Recommendations
### Immediate Actions
1. **Identify Critical Service Configuration:** Locate the Pluggable Authentication Module (PAM) configuration files (e.g., `/etc/pam.d/sshd` for SSH) on all target Linux systems that utilize PAM for authentication.
2. **Install Necessary PAM Modules:** If using the `pam_script` method detailed, download, compile, and install the `pam_script` module.
* Clone the repository (e.g., from `https://github.com/jeroennijhof/pam_script`).
* Compile and install using standard compilation tools (`autoreconf -i`, `./configure`, `make`, `make install`).
3. **Configure PAM to Call Script:** Inject the necessary line into the relevant PAM configuration file (e.g., `/etc/pam.d/sshd`) to call the monitoring script upon authentication events.
* **Action:** Add `auth optional pam_script.so` to the chosen PAM stack configuration.
### Short-term Improvements (1-3 months)
1. **Implement Canary Credential Checks:** Develop and deploy monitoring scripts hooked via PAM that check authentication variables against a predefined list of known compromised credentials or specific canary credentials.
2. **Integrate External Alerting Mechanism:** Configure the monitoring script to trigger an external notification service upon detection.
* **Action:** Use minimalist HTTP requests (like `curl`) to trigger a service, such as Canary Tokens, which handles external alerting (email, Slack, etc.) without requiring on-premise mail infrastructure setup in the script itself.
3. **Develop Credential Storage Strategy:** Move hardcoded plaintext credentials out of the authentication script and into a more secure location (e.g., file with restricted permissions or a memory-resident structure).
* **Guidance:** Consider hashing the canary passwords within the script and comparing them against stored hashes, or passing values for verification to a dedicated, hardened local service.
### Long-term Strategy (3+ months)
1. **Expand Monitoring Scope:** Extend detection logic beyond SSH to cover other PAM-dependent services (e.g., graphical logins like XSession, `sudo`, or local console logins) by updating the corresponding configuration files in `/etc/pam.d/`.
2. **Develop Custom PAM Module (Advanced):** For higher security and stealth, investigate developing a lightweight, custom PAM module instead of relying on older third-party scripts like `pam_script`, which may not be actively maintained in modern distributions.
3. **Establish Centralized Logging and Audit:** Integrate the notification trigger mechanism with a Security Information and Event Management (SIEM) system for historical analysis, correlation, and long-term compliance auditing, rather than relying solely on token alerts.
## Implementation Guidance
### For Small Organizations
* **Focus on SSH:** Immediately deploy the `pam_script` solution targeting the `/etc/pam.d/sshd` configuration on all primary access servers to cover the highest risk vector (remote administrative access).
* **Use Simple Tokens:** Leverage basic Canary Tokens configured to send immediate emails for rapid, low-infrastructure alerting.
### For Medium Organizations
* **Standardize PAM Configuration:** Implement configuration management tools (e.g., Ansible, Puppet) to ensure the PAM hook (`pam_script.so` call) and the corresponding script are consistently deployed across all servers.
* **Inventory Compromised Credentials:** Systematically collect and catalog known breached credentials discovered from external reports or threat intelligence feeds to use as canary values.
### For Large Enterprises
* **Audit PAM Dependencies:** Conduct a full audit of all services relying on PAM. Determine if custom module development is necessary to meet performance or security requirements, especially in high-volume environments.
* **Integrate with Identity Management:** Ensure that the verification step (even if triggered by the canary) is ultimately reconciled against a central identity provider (IdP) for standardized response actions (e.g., automatic account quarantine).
* **Secure Script Execution:** Use SELinux or AppArmor profiles to strictly limit what the monitoring script can access and execute, mitigating risk if the script itself becomes a vector.
## Configuration Examples
The core mechanism involves checking environment variables provided by PAM.
bash
#!/bin/sh
# Available Env variables provided by PAM:
# PAM_SERVICE - the application that's invoking the PAM stack (e.g., sshd)
# PAM_TYPE - the module-type (e.g. auth, account, session, password)
# PAM_USER - the user being authenticated into
# PAM_AUTHTOK - password in readable text (HIGHLY SENSITIVE!)
CANARY_USER="service_account_01"
CANARY_PASS="ThePoliceHaveMe"
CANARY_URL="http://canarytokens.com/articles/n6p9nzmbq4o1o19vmvzxuqg8m/index.html"
# Check if the attempting user matches the canary user AND the password matches the canary password
if [ "$PAM_USER" = "$CANARY_USER" ] && [ "$PAM_AUTHTOK" = "$CANARY_PASS" ]; then
# Trigger the notification token
curl "$CANARY_URL"
fi
# Exit 0 indicates success to the PAM stack, allowing the login to proceed (covert operation)
exit 0
## Compliance Alignment
This implementation directly supports controls related to:
* **NIST SP 800-53 (AC-7):** Audit and Accountability – Establishing mechanisms to track user sessions and activity reporting unauthorized access attempts.
* **ISO/IEC 27001 (A.12.4.1):** Event Logging and Monitoring – Implementing technical monitoring to detect abnormal or unauthorized activity.
* **CIS Controls (Control 12):** Data Recovery and Monitoring (specifically relating to unauthorized access detection).
* **PCI DSS (Requirement 10.2.3):** Logging system activity, including all access control mechanisms.
## Common Pitfalls to Avoid
1. **Relying on Plaintext Passwords Long-Term:** Storing the canary password in plaintext within the script is acceptable for demonstration but constitutes a significant risk if the script itself is compromised before the token is triggered. Remediate this by hashing or using a verified challenge/response mechanism.
2. **Failing to Restart Services:** Not restarting the relevant service (e.g., `sshd`) after modifying the PAM configuration might result in the new module not being loaded immediately.
3. **Ignoring PAM Stack Order:** Placing the `auth optional pam_script.so` line in the wrong position within the PAM stack could cause it to execute too late, too early, or interfere unexpectedly with legitimate authentication methods. Generally, authorization checks (like this one) are placed early in the `auth` stack.
4. **Failure to Handle Non-Linux Systems:** The method described relies heavily on the Linux PAM structure. Do not assume this implementation details apply directly to Windows or macOS without finding equivalent authentication hooks.
## Resources
* **Canary Tokens Documentation:** The service used for non-infrastructure alerting (search for "Canary Tokens").
* **PAM Documentation:** Official documentation on Pluggable Authentication Modules configuration for deep understanding of stack order and environment variables.
* **`pam_script` Repository:** Source code reference for the specific module used in the implementation (e.g., `https://github.com/jeroennijhof/pam_script`).