Full Report
How IBM Cloud caught us exploring its infrastructure and how a hardcoded secret eventually led to build artifact access and manipulation
Analysis Summary
# Vulnerability: Hell’s Keychain - Supply-Chain Attack Vector in IBM Cloud Databases for PostgreSQL
## CVE Details
- CVE ID: Not explicitly provided in the text. (Note: Cloud service specific vulnerabilities often lack immediate public CVEs; the reference points to an IBM advisory.)
- CVSS Score: Not explicitly provided in the text.
- CWE: Likely related to Improper Control of Generation of Code or Data (CWE-94) concerning the build process integrity, and multiple secret management weaknesses.
## Affected Systems
- Products: IBM Cloud Databases for PostgreSQL.
- Versions: Not specified, but the vulnerability was present in the managed PostgreSQL service environment.
- Configurations: Exploitation required an initial foothold (e.g., via PostgreSQL privilege escalation) combined with an overly permissive network link between the customer environment and internal IBM Cloud build servers.
## Vulnerability Description
"Hell’s Keychain" is a novel, supply-chain vulnerability characterized by a chain of three exposed secrets coupled with overly permissive network access (the "forbidden link") between a customer-facing PostgreSQL database environment and internal IBM Cloud build servers.
The "Keychains" consisted of:
1. A Kubernetes service account token.
2. A private container registry password.
3. CI/CD server credentials.
When combined, these secrets, accessible via an initial privilege escalation within the PostgreSQL instance (potentially via SQL Injection leveraging Logical Replication), allowed the attacker to authenticate to IBM Cloud’s internal build servers. This could enable remote code execution in customer environments, the reading/modification of data in the PostgreSQL database, and intervention in IBM Cloud’s internal image building process, leading to a potential supply-chain attack impacting customers.
## Exploitation
- Status: Security research demonstrated successful chaining of vulnerabilities (PoC available internally/through research disclosure). IBM Cloud stated **no indication that systems were exploited further or by other parties** prior to patching.
- Complexity: High (Requires chaining multiple distinct weaknesses: initial privilege escalation, secret discovery, and network access confirmation).
- Attack Vector: Network (Once initial access to PostgreSQL is achieved, the chain uses network access to internal resources).
## Impact
- Confidentiality: High (Access to internal secrets, modification of customer data).
- Integrity: High (Ability to manipulate artifacts in the CI/CD build process, potentially injecting malicious code into customer images).
- Availability: Medium (While not a direct denial of service, impact on the integrity of the build process affects service reliability).
## Remediation
### Patches
- IBM Cloud fully mitigated all reported issues as of September 3, 2022.
- **No customer action is required** as the fix was applied server-side by IBM Cloud.
### Workarounds
- Since IBM Cloud fully patched the environment for all customers, no customer-side workarounds were necessary or specified.
## Detection
- Indicators of compromise would likely involve unauthorized access to internal IBM Cloud build servers, unusual network flows between production and build environments, or manipulation of artifacts pushed to internal registries.
- Detection focuses on monitoring the service provider's internal cloud infrastructure security posture, specifically secrets exposure within the PostgreSQL container environment and overly permissive network policies allowing egress to CI/CD infrastructure.
## References
- Vendor Advisory: hxxps://ibm.com/support/pages/node/6842111
- Research Database Entry: hxxps://www.cloudvulndb.org/hellskeychain
- Related Research: hxxps://www.wiz.io/blog/the-cloud-has-an-isolation-problem-postgresql-vulnerabilities