Full Report
Cybercriminals are increasingly gravitating towards uncensored LLMs, cybercriminal-designed LLMs and jailbreaking legitimate LLMs.
Analysis Summary
# Threat Actor: Cybercriminals Utilizing LLMs (General Trend)
## Attribution & Identity
This summary refers to a broad trend of **cybercriminals** leveraging Artificial Intelligence (AI) technologies, specifically Large Language Models (LLMs). No single named, attributed threat actor is definitively identified as the primary focus, but specific *prolific users* of dark web LLM services are mentioned.
**Known Aliases and Associated Groups:**
* **CanadianKingpin12:** Developer/marketer of **FraudGPT**.
* **FraudGPT (Tool/Platform):** Marketed on the dark web and Telegram.
## Activity Summary
Cybercriminals are actively integrating LLMs into their attack lifecycle to streamline and enhance criminal hacking activities. This includes:
1. **Using Uncensored LLMs:** Employing models like OnionGPT or **Llama 2 Uncensored** (run via frameworks like Ollama) that lack alignment and guardrails.
2. **Utilizing Specialized Criminal LLMs:** Subscribing to custom-built malicious LLMs (e.g., FraudGPT, WormGPT, DarkGPT) advertised on the dark web.
3. **Abusing Legitimate LLMs:** Employing jailbreaking techniques against aligned models (like ChatGPT) to bypass safety filters.
4. **Malmodel Distribution:** Distributing backdoored AI models via platforms like Hugging Face, where malicious Python code is embedded in serialized files (Pickle format) to execute upon deserialization.
5. **Exploiting RAG Systems:** Manipulating Retrieval Augmented Generation (RAG) databases to poison lookup results and influence LLM output.
## Tactics, Techniques & Procedures
- **Evasion/Bypassing Controls:** Jailbreaking legitimate LLMs to circumvent safety mechanisms and guardrails.
- **Tool Generation:** Using LLMs to write malicious code, create undetectable malware, and generate hacking tools or phishing content.
- **Phishing/Scam Generation:** Creating convincing phishing emails, scam pages, and SMS messages at scale.
- **Infrastructure Setup:** Utilizing LLM features for automatic script creation (e.g., replicating logs/cookies) and utilizing embedded page hosting capabilities.
- **Vulnerability Scanning:** Using LLMs integrated with CVE databases (PRO feature of FraudGPT) to scan websites for vulnerabilities.
- **Supply Chain Compromise (AI Models):** Embedding malicious execution code within serialized AI model files (Pickle objects) downloaded by users.
- **Data Poisoning:** Manipulating RAG databases to force specific malicious outputs from an LLM.
## Targeting
The tooling suggests broad targeting capability, though specific victim types are implied by the advertised features:
* **Sectors:** General cybersecurity, financial fraud (CVV checking, bin finding), and software development/infrastructure (vulnerability scanning).
* **Geography:** Not explicitly defined, but tools suggest global reach (e.g., finding cardable sites, generating global phishing campaigns).
* **Victims:** Entities susceptible to phishing, malware infection, or organizations hosting RAG systems. Specific named victim organizations were not detailed in the summary.
## Tools & Infrastructure
**Malware/LLM Families Used:**
* OnionGPT (Uncensored)
* Llama 2 Uncensored (via Ollama)
* WhiteRabbitNeo
* FraudGPT
* WormGPT
* DarkGPT
* DarkestGPT
* GhostGPT
**Infrastructure:**
* **Hugging Face:** Platform used for distributing backdoored (Pickle) models.
* **Dread, Telegram:** Forums/platforms used to advertise and facilitate access to malicious LLMs (e.g., OnionGPT, FraudGPT).
* **GoldCheck API:** Utility advertised as integrated into FraudGPT for CVV checking.
## Implications
The integration of LLMs acts as a **force multiplier** for cybercriminals. It significantly reduces the barrier to entry for complex attacks, enhances the quality and stealth of generated assets (phishing content, malware code), and automates parts of the attack lifecycle previously requiring manual expertise. The rise of uncensored and custom LLMs bypasses controls established by major AI vendors.
## Mitigations
- **Source Verification:** Exercise extreme caution when downloading and running AI models, especially from untrusted sources. Ensure files come from trusted providers.
- **Sandboxing:** Run downloaded AI models (particularly those relying on dangerous serialization formats like Pickle) in an isolated sandbox environment before deployment.
- **Model Scanning:** Utilize security tools like Picklescan to regularly inspect uploaded models for malicious code, recognizing that these tools may have vulnerabilities.
- **Input Validation for RAG:** Implement rigorous validation and auditing processes for external data sources feeding Retrieval Augmented Generation (RAG) systems to prevent data poisoning.
- **Operational Security for LLM Users:** Recognize that using jailbreaking techniques or custom LLMs potentially exposes users to detection by threat intelligence focused on these emerging criminal communities.