Full Report
Some misconfigured AI chatbots are pushing people’s chats to the open web—revealing sexual prompts and conversations that include descriptions of child sexual abuse.
Analysis Summary
# Incident Report: Exposure of Sensitive Prompts in Open-Source AI Chatbots
## Executive Summary
Security researchers discovered that numerous AI chatbots, primarily those using the open-source `llama.cpp` framework for role-playing and sexual conversation, were leaking user prompts to the public internet in near real-time due to misconfigurations. The exposed data included highly sensitive content, notably detailed scenarios involving child sexual abuse material (CSAM). The incident was discovered via active web scanning, and while no direct user PII was exposed, the exposure highlights severe risks associated with improperly deployed generative AI systems.
## Incident Details
- **Discovery Date:** March (Date not specified, but discovery scanning occurred this month)
- **Incident Date:** Ongoing exposure leading up to and during the discovery period.
- **Affected Organization:** Various small instances/deployments using `llama.cpp`; no single organization identified.
- **Sector:** Technology/Generative AI Services (Likely consumer-facing role-playing applications).
- **Geography:** Global (Data collected included prompts in English, Russian, French, German, and Spanish).
## Timeline of Events
### Initial Access
- **Date/Time:** Pre-March/Ongoing. Attackers or users utilized misconfigured AI systems running `llama.cpp`.
- **Vector:** Misconfiguration of open-source AI deployments (specifically using the `llama.cpp` framework).
- **Details:** Improper setup allowed prompts sent to the AI models to be inadvertently exposed publicly.
### Lateral Movement
- Not explicitly detailed, as the attack vector focused on data exfiltration via configuration error rather than traditional network compromise. The exposure was due to insecure system output visibility.
### Data Exfiltration/Impact
- **What was stolen or damaged:** Approximately 1,000 user prompts collected over 24 hours across 117 identified leaking IP addresses. Content included highly explicit sexual role-playing scenarios, including five scenarios involving children as young as 7.
### Detection & Response
- **How it was discovered:** Researchers from the security firm UpGuard actively scanned the web for misconfigured AI systems in March.
- **Response actions taken:** UpGuard collected data for 24 hours to analyze the scope. The article indicates that a separate, South Korea-based image generator implicated in similar issues was shut down after being approached by WIRED, but specific response actions for the 400 leaked `llama.cpp` instances are not detailed other than the research observation.
## Attack Methodology
- **Initial Access:** **Misconfiguration.** Attackers/users accessed the chat functionality of improperly configured AI instances.
- **Persistence:** N/A (This was a configuration vulnerability, not a persistence mechanism established by an external threat actor).
- **Privilege Escalation:** N/A
- **Defense Evasion:** N/A (The issue stemmed from lack of security configuration, not active evasion techniques).
- **Credential Access:** N/A (No usernames or PII were found in the leaked data).
- **Discovery:** N/A (The discovery was external scanning by security researchers).
- **Lateral Movement:** N/A
- **Collection:** User prompts were collected by the AI system and subsequently leaked to public endpoints.
- **Exfiltration:** Data was exfiltrated via insecure configuration revealing prompts almost instantaneously to the web.
- **Impact:** Exposure of highly sensitive personal fantasies, raising legal and ethical concerns regarding CSAM interaction.
## Impact Assessment
- **Financial:** Not disclosed.
- **Data Breach:** Hundreds of sensitive role-playing prompts, spanning multiple languages, including descriptions of child sexual abuse scenarios (approx. 952 messages analyzed). No identifying personal information (usernames/PII) was exposed.
- **Operational:** Minimal direct operational impact on the organizations hosting the systems, as most appeared to be amateur/test setups. However, significant reputational and legal risk for services deploying unregulated LLMs.
- **Reputational:** High reputational risk for the technology sector due to the exposure of illicit content generated via these AI tools.
## Indicators of Compromise
- **Network indicators:** Identified 117 IP addresses leaking prompts (specific IPs defanged/withheld).
- **File indicators:** N/A (The 'leak' was data output, not malicious binaries).
- **Behavioral indicators:** Real-time, unprotected output streams from generative AI inference endpoints using `llama.cpp`.
## Response Actions
- **Containment measures:** Researchers collected samples to quantify the issue. The underlying cause requires securing the configuration of the `llama.cpp` deployments.
- **Eradication steps:** Not detailed as to whether the specific 400 discovered instances were remediated by their owners.
- **Recovery actions:** In the case of the related image generator, the service was shut down by the operator after contact.
## Lessons Learned
- **Key takeaways:** Open-source AI frameworks like `llama.cpp` enable rapid deployment but carry significant risk if configuration security is overlooked. These tools are actively being used globally for illicit activities, including the generation of material related to CSA fantasies.
- **What could have been done better:** Regulatory oversight is severely lacking, failing to match the rapid deployment reality of generative AI technology. Deployers failed to properly secure the output channels of their AI environments.
## Recommendations
- Implement mandatory security configurations, validation, and auditing for all production and staging environments utilizing user-facing generative AI components.
- Organizations utilizing open-source LLM frameworks must ensure output data streams are secured and not publicly accessible by default.
- Urgently address regulatory gaps concerning the use of generative AI to simulate or create scenarios involving child sexual abuse.