Full Report
In the hands of malicious actors, AI tools can enhance the scale and severity of all manner of scams, disinformation campaigns and other threats
Analysis Summary
# Main Topic
The escalating threat posed by malicious actors leveraging Artificial Intelligence (AI) and Generative AI (GenAI) tools to significantly enhance the scale, severity, and sophistication of cyber threats, primarily focusing on scams, social engineering, and disinformation campaigns.
## Key Points
- AI is expected to increase the volume and impact of cyberattacks, particularly in the next two years (as warned by the UK NCSC).
- The most acute threat area is social engineering, where GenAI enables the crafting of highly convincing campaigns in flawless local languages.
- AI is being used to automate large-scale reconnaissance for vulnerable asset identification.
- Specific emerging threats leveraging AI tools in 2025 include:
- **Authentication Bypass:** Use of deepfake technology to impersonate customers for new account creation or access validation.
- **Business Email Compromise (BEC):** Sophisticated social engineering campaigns utilizing deepfake audio/video to impersonate CEOs/senior leaders in calls and virtual meetings to trick recipients into wiring funds.
- **Impersonation Scams:** Training open-source LLMs on scraped social media data to impersonate victims for virtual kidnapping or other scams targeting friends and family.
- **Influencer Scams:** Creation of fake/duplicate social media accounts mimicking celebrities via GenAI and deepfake video to lure followers into crypto or investment scams.
- **Disinformation:** Hostile states using GenAI to easily generate fake content to influence users and create harder-to-detect influence operations compared to traditional troll farms.
- **Password Cracking:** AI-driven tools capable of unmasking user credentials in seconds for mass access to corporate networks and customer accounts.
- There is an associated risk of data leakage where sensitive data (biometrics, financials) used to train LLMs could be compromised or inadvertently shared by corporate users via GenAI prompts.
## Threat Actors
- Every type of threat actor is confirmed to be using AI tools.
- Hostile states are explicitly mentioned as tapping GenAI for advanced disinformation operations.
- Fraudsters and scammers are leveraging LLMs and deepfake technology for financial fraud and impersonation.
## TTPs
- Mass generation of convincing content in multiple languages (Social Engineering/Disinformation).
- Automated large-scale identification of vulnerable assets (Reconnaissance).
- Use of Deepfake Audio/Video for identity deception (Impersonation/BEC).
- Training open-source LLMs on scraped social media data for targeted impersonation.
- AI-driven credential harvesting and password cracking at scale.
- (Note: Specific MITRE ATT&CK IDs were not provided in the source, but techniques align broadly with T1566 Phishing/Social Engineering and T1027 Obfuscation.)
## Affected Systems
- Corporate networks and customer accounts (via password cracking).
- Authentication systems relying on selfie/video checks.
- Corporate email/communication infrastructure (BEC targets).
- Social media platforms (targets of impersonation and source of training data).
## Mitigations
- Increased vigilance required from end-users, especially regarding impersonation (CEO fraud, influencer scams).
- Social media platforms need to develop and deploy effective account verification tools and badges.
- Corporate security teams must understand AI limitations and maintain human expertise in the decision-making process to mitigate AI hallucinations and errors.
- Organizations must address the risk of sensitive corporate data leakage via employee use of GenAI applications (one poll noted a fifth of UK CISOs saw staff accidentally expose data via GenAI use).
- The need for collaboration between governments, enterprises, and end-users to manage these evolving risks.
## Conclusion
AI represents a paradigm shift in modern cyber threats, moving defenses into an arms race where both defenders and attackers utilize the technology. The immediate focus must be on hardening social engineering defenses, verifying digital identities (voice/video), and establishing clear corporate policies regarding the use of GenAI tools to prevent unintentional data exposure. Relying solely on current security stacks without integrating AI-aware defenses and strong human oversight will prove insufficient in 2025.