Full Report
AI coding assistants can hallucinate non-existent package names. According to studies, somewhere between 5-21% are bad. 58% of these were recurring across multiple sessions. In the past, typosquatting was a thing: create packages with common misspellings of a package to install code that the developer did not intend to install. This is the same thing but for AI generated code. To do this, monitor the AI output for commonly hallucinated package names, install the packages on PyPI, npm, RubyGems and other locations. By doing this, you can trick AI generated code to install malicious packages. This is website is a tool that provides a lazy-loading trust scoring system with different tiers. With tier one, it does download, age and version history scoring. In tier 2 it does dependency analysis. In tier 3, it does a deep analysis of maintainer reputation, email domain verification and GitHub detection. This claims to have a very good accuracy and solve the problem. Under the hood, it's just a simple open source tool that does the scanning and even has GitHub CI integration support. Pretty neat!
Analysis Summary
# Best Practices: AI Hallucination & Supply Chain Security
## Overview
These practices address "AI Package Hallucination," a new attack vector where AI coding assistants (ChatGPT, Claude, Copilot) suggest non-existent software packages. Attackers monitor these hallucinations and register malicious packages under those names. These guidelines also mitigate traditional typosquatting, namespace squatting, and automated supply chain attacks.
## Key Recommendations
### Immediate Actions
1. **Validate AI-Generated Imports:** Treat every library or package suggested by an AI assistant as "untrusted" until its existence and reputation are verified in a public registry (npm, PyPI, RubyGems).
2. **Deploy SlopGuard:** Use an automated scanning tool to check Software Bill of Materials (SBOM) for low-reputation or non-existent packages.
3. **Audit Version Spikes:** Manually review any package that has published more than 5 versions in 24 hours (a high-severity signal for automated malware distribution).
### Short-term Improvements (1-3 months)
1. **Incorporate "Lazy-Loading" Trust Scoring:** Implement a tiered scanning system (Downloads -> Dependency Analysis -> Deep Reputation) to reduce API overhead while maintaining 99%+ accuracy.
2. **CI/CD Integration:** Gate all production builds by requiring an SBOM scan. Set the CI/CD pipeline to fail (Exit Code 1) if high-risk or "untrusted" packages (score <40) are detected.
3. **Email Domain Verification:** Implement checks to ensure packages claiming to be from a specific organization (e.g., Google, Amazon) are maintained by an email address within that org's verified domain.
### Long-term Strategy (3+ months)
1. **Layered Defense-in-Depth:** Integrate AI hallucination detection with behavioral analysis (Socket), CVE detection (Snyk/Dependabot), and manual code review.
2. **Ecosystem Monitoring:** Establish a process for tracking maintainer ownership changes to prevent account takeover attacks from going unnoticed.
3. **Internal Allowlisting:** Build a curated list of "known-good" internal and community packages to bypass scanning for trusted entities and speed up build times.
## Implementation Guidance
### For Small Organizations
- **Manual Scripting:** Run `slopguard` locally before committing AI-generated code.
- **Focus on Stage 1 Signals:** Prioritize Download count (>100k) and Age (>2 years) as these two metrics alone catch 99.5% of malicious packages.
### For Medium Organizations
- **CI Integration:** Standardize GitLab or GitHub Actions to scan `sbom.json` on every Merge Request.
- **GitHub Token Management:** Use an authenticated `GITHUB_TOKEN` to increase API rate limits (5,000 req/hr) for deep reputation analysis.
### For Large Enterprises
- **Multi-Scanner Orchestration:** Use SlopGuard for hallucination/typosquatting and Snyk for CVEs.
- **Namespace Protection:** Verify that internal package names do not overlap with public registry names to prevent "Dependency Confusion" attacks.
## Configuration Examples
### GitLab CI/CD Configuration
Add this job to your `.gitlab-ci.yml` to block builds containing suspicious AI-generated dependencies:
yaml
slopguard_scan:
stage: test
image: ruby:3.1
script:
- git clone hxxps://github[.]com/aditya01933/SlopGuard.git
- cd SlopGuard && bundle install && cd ..
- ./SlopGuard/slopguard sbom.json --format gitlab --output report.json
artifacts:
reports:
dependency_scanning: report.json
allow_failure: false # Exit code 1 will stop the pipeline
### Trust Scoring Thresholds
| Score | Classification | Security Action |
| :--- | :--- | :--- |
| **80-100** | High Trust | Automated Approval |
| **60-79** | Medium Trust | Warning / Info Scan |
| **40-59** | Low Trust | Full Deep Deep Analysis Required |
| **<40** | Untrusted | **Block Installation & Notify Security** |
## Compliance Alignment
- **NIST SSDF (Secure Software Development Framework):** Aligns with "Produce Secure Software" by verifying third-party components.
- **ISO 27001 (A.14.2.2):** Addresses system change control procedures and dependency validation.
- **CIS Controls (Control 7):** Vulnerability Management and software inventory control.
## Common Pitfalls to Avoid
- **The "Popularity Trap":** Assuming a package is safe just because it has downloads. Always check "Version Spikes" and "Ownership Changes."
- **Offline Scanning Limitations:** Tools like SlopGuard require internet access for registry APIs; they cannot validate hallucinated names in air-gapped environments without local registry mirrors.
- **Neglecting Internal Namespaces:** Attackers can "squat" on names that sound like your internal company libraries. Use an allowlist for your own `@org/package` names.
## Resources
- **SlopGuard Repository:** `hxxps://github[.]com/aditya01933/SlopGuard`
- **Behavioral Analysis:** `hxxps://socket[.]dev`
- **Vulnerability Database:** `hxxps://snyk[.]io`
- **SBOM Generation:** `hxxps://cyclonedx[.]org`