Full Report
AI creates what it’s told to, from plucking fanciful evidence from thin air, to arbitrarily removing people’s rights, to sowing doubt over public misdeeds.
Analysis Summary
# Main Topic
The primary threat centers on the significant security and reliability risks posed by human misuse of current Artificial Intelligence (AI) technologies, rather than imminent threats from Artificial General Intelligence (AGI). This misuse includes the generation of fabricated information, the arbitrary removal of rights based on AI actions, and the deliberate sowing of doubt regarding public matters. A key element of documented misuse is the over-reliance on AI tools, leading to demonstrable failures in professional contexts.
## Key Points
- The major risk in the near term (2025/2026 outlook) is human misuse, not advanced AGI capabilities.
- AI models, such as large language models (LLMs), exhibit tendencies to "make stuff up" (hallucinate), which users fail to vet.
- This failure manifests as AI creating entirely fanciful evidence or false legal precedents.
- The implications extend beyond technical errors to actual harm, such as arbitrary removal of rights or the erosion of public trust.
## Threat Actors
- **Threat Actors:** Professionals (e.g., lawyers) who are over-relying on public AI tools without adequate verification of outputs.
- **Motivation:** A combination of lack of awareness regarding AI limitations and over-trust in modern tools.
- **Attribution:** Individual professionals acting in self-interest or negligence, leading to documented sanctions across various jurisdictions.
## TTPs
- **Technique:** Using generative AI (like ChatGPT) to produce content (e.g., legal briefings, case citations) intended for official submission.
- **Failure Mode:** Accepting AI-generated hallucinations (fictitious evidence or non-existent legal cases) as factual.
- **Impact TTP:** Deploying falsified outputs in official proceedings, leading to professional sanctions and erroneous legal positioning.
## Affected Systems
- **Systems:** General-purpose large language models (LLMs) utilized by end-users (e.g., ChatGPT).
- **Affected Sectors:** Primarily the legal profession, demonstrated by cases in:
- British Columbia
- New York
- Colorado
- **Scope:** Any professional workflow where unverified AI-generated output is incorporated into final deliverables.
## Mitigations
- **Verification Imperative:** Users must actively verify all factual claims, citations, and evidence generated by AI tools before professional deployment.
- **User Awareness:** Training professionals on the known limitations of current generative AI, specifically its tendency to hallucinate facts.
- **Accountability:** Establishing clear lines of accountability, as demonstrated by the courts holding individuals (lawyers) responsible for AI-generated errors.
## Conclusion
The immediate threat landscape emphasizes the critical deficiency in **human oversight** of AI results. Organizations must urgently implement policies requiring rigorous validation of all factual and evidential material produced by generative AI to prevent professional misconduct, erroneous decision-making, and downstream societal harm stemming from fabricated realities. Unintentional misuse (over-reliance) is currently a more tangible threat than malicious exploitation by sophisticated adversaries.