Full Report
CI/CD pipelines, as an essential part of the software development process, are an attractive target to malicious actors. Based on our research of cloud environments, we share common misconfigurations and provide tips on how to remediate them in order to prevent supply-chain attacks.
Analysis Summary
# Best Practices: Hardening CI/CD Pipelines Against Software Supply-Chain Attacks
## Overview
These practices address common, often overlooked misconfiguration patterns within CI/CD pipelines that expose cloud environments to software supply-chain attacks, enabling adversaries to achieve lateral movement and tamper with artifacts. The focus is on mitigating risks associated with overprivileged access, forgotten secrets, and weak network isolation between development and production components.
## Key Recommendations
### Immediate Actions
1. **Audit and Inventory Secrets:** Immediately scan all internet-facing CI/CD components (servers, logs, build agents) for forgotten secrets, credentials (especially cloud access keys), or sensitive data stored in easily accessible locations (e.g., `.bash_history` files or environment variables).
2. **Review Container Registry Access:** Immediately scope down all existing container registry permissions. Ensure identities interacting with registries are restricted to the absolute minimum required privilege level (`read-only` or `write-only`).
3. **Disable Resource Listing Permissions:** Revoke the `listing resources` permission for any automated application or CI/CD process accessing container registries to prevent unauthorized reconnaissance of accessible repositories.
### Short-term Improvements (1-3 months)
1. **Implement Least Privilege for Registry Interactions:** Explicitly enforce that only the final stage of the pipeline (the deployment step) should possess `push` permissions to the container registry. Developers should not have direct write access.
2. **Enforce Network Isolation:** Implement strong network policies and cloud segmentation to isolate critical assets (like storage buckets holding production artifacts) away from general CI/CD virtual networks or internet-facing servers.
3. **Audit Cloud Key Exposure:** Monitor cloud access keys present in internet-facing environments and verify they do not grant unauthorized access to CI/CD environments or storage buckets.
### Long-term Strategy (3+ months)
1. **Adopt Robust Secret Management Solutions:** Migrate from storing secrets in file systems or configuration files to dedicated, centralized secrets management vaults integrated securely with the CI/CD tooling.
2. **Mandate Network Controls for Critical Assets:** Ensure that access to critical cloud identities and storage buckets is strictly restricted to authorized CI/CD Virtual Private Networks (VPNs) or dedicated cloud segments.
3. **Strengthen SaaS CI/CD Security:** For SaaS solutions (like GitHub workflows), where network scoping is difficult, enforce mandatory **Role-Based Access Controls (RBAC)** and **Two-Factor Authentication (2FA)** for all users and service accounts interacting with the platform.
## Implementation Guidance
### For Small Organizations
- Focus primarily on the **Immediate Actions**: conducting thorough secret audits and strictly limiting container registry roles to read/write only where absolutely necessary.
- Utilize cloud-native IAM capabilities for access scoping as a primary control mechanism.
- Enforce 2FA on all developer and administrator accounts related to version control and CI/CD tooling.
### For Medium Organizations
- Prioritize **Short-term Improvements**: establish clear network segmentation strategies between build infrastructure and artifact storage/production environments.
- Begin adopting cloud identity federation rather than using long-lived static credentials (cloud keys) within CI/CD runners.
- Document and enforce role-specific permissions for registry access roles.
### For Large Enterprises
- Implement a **Long-term Strategy** focusing on enterprise-scale secrets management platforms integrated across multiple cloud providers.
- Establish formal Cloud Security Posture Management (CSPM) tooling to continuously monitor for credential exposure in internet-facing assets and unsegmented network access policies.
- Design and enforce immutable infrastructure principles to minimize the risk of compromised build agents persisting secrets.
## Configuration Examples
*Note: Specific technical configuration snippets were not provided in the source text, but the principles emphasize the following:*
1. **Container Registry IAM Policy Snippet Focus:** Policies must explicitly deny the `ecr:DescribeRepositories` or equivalent actions unless the role is an administrative role, thereby preventing reconnaissance via resource listing for standard build roles.
* *Goal:* Restrict identity permissions to specific actions (`ecr:BatchCheckLayerAvailability`, `ecr:GetDownloadUrlForLayer`) on specific resources (repository ARNs).
2. **Network Security Group/Firewall Rule Focus:** Implement ingress rules on critical storage assets (e.g., S3 buckets, Azure Storage) that only permit traffic sourced from the specific internal subnet ranges dedicated to the CI/CD execution environment.
## Compliance Alignment
The recommended practices strongly align with foundational principles across major security standards:
* **NIST Cybersecurity Framework (CSF):** Directly supports **Identify** (Asset Management, Risk Assessment) and **Protect** (Access Control).
* **CIS Benchmarks:** Addresses controls related to Identity and Access Management (IAM) and Network Security Configuration.
* **ISO/IEC 27001 (A.9 Access Control & A.14 System Acquisition, Development & Maintenance):** Mandates securing system components and enforcing least privilege.
## Common Pitfalls to Avoid
1. **Believing Network Controls are Sufficient:** Relying solely on VPNs or VNETs to protect secrets in internet-facing applications, as a compromise of that application bypasses all network restrictions when the secret is directly accessible.
2. **Over-Permissive Developer Roles:** Granting developers or base build images write/push access to production container registries.
3. **Leaving Artifacts Accessible via Default Settings:** Failing to review the default permissions on cloud storage buckets where artifacts or build outputs are placed, which can often grant broad access if not explicitly hardened.
4. **Ignoring Obscure Logs:** Overlooking simple data remnants like the contents of `.bash_history` or shell execution logs where credentials might have been temporarily typed or exposed during the image build process.
## Resources
- **Cloud Provider IAM Documentation:** Consult documentation for fine-grained access control policies specific to container registry and storage services.
- **Secrets Management Platform Documentation:** Evaluate solutions for centralized, vault-based secret injection into CI/CD runners (e.g., HashiCorp Vault, AWS Secrets Manager, Azure Key Vault).
- **CIS Controls:** Review sections related to secure configuration of development tools and infrastructure as code principles.