Full Report
Learn why your existing security tech won’t detect data exposure, prompt injection and manipulation, and other AI security risks from ChatGPT Enterprise, Microsoft 365 Copilot, and other LLMs.Key takeaways:Even when they’re approved, generative AI tools like Microsoft Copilot and ChatGPT create an entirely new attack surface for organizations that behaves differently from the traditional IT attack surface and even the cloud attack surface. Enterprise AI tools can instantly connect to your organization’s most sensitive data depending on how they’re configured. Clever employees can trick generative AI tools into exposing strategic plans, mergers and acquisitions, salary information, and other sensitive data. Due to the uniqueness of the AI attack surface and the scale, ubiquity, and agency (ability to take autonomous action) of enterprise AI tools, organizations can’t rely on traditional security tools like data loss prevention software (DLP), cloud access security brokers (CASBs), or cloud security posture management (CSPM) to protect them from generative AI security risks.When it comes to securing AI, many organizations believe they can mitigate enterprise AI risks by prohibiting employees from using public AI tools and instead directing them to use approved enterprise AI tools.While enterprise versions of generative AI tools can help keep sensitive data out of public large language models (LLMs), the risk of data exposure remains. Why? Because approved, enterprise AI tools can instantly connect to your organization’s most sensitive data when implemented, depending on how they’re configured.I repeat: When implemented, enterprise AI tools — including ChatGPT Enterprise, Microsoft Copilot, and others — can instantly connect to systems, such as your enterprise resource planning (ERP), human resources (HR), and customer relationship management (CRM) tools, where confidential customer data, employee data, financial data, intellectual property, and strategic plans reside.As a result, employees can prompt enterprise generative AI tools to share sensitive data when these tools are configured to integrate with systems containing sensitive data. And if employees can do it, threat actors can, too.I’ve seen many examples of employees finding creative ways to manipulate enterprise AI chatbots into sharing confidential data about equity deals, non-disclosure agreements, performance appraisals, financial projections, board slides, and more. If preventing this kind of rogue behavior, which creates an insider threat, isn’t part of your organization’s AI acceptable use policy and training, it should be.I expect to see threat actors leverage misconfigurations in AI tools to access sensitive data and use prompt injection attacks to instruct AI agents to divert payments to cyber criminals’ bank accounts.Your existing security tools — specifically, your data loss prevention (DLP), cloud access security brokers (CASBs), and cloud security posture management (CSPM) — won’t detect or prevent AI security risks because they were built to monitor a very different attack surface.The unique challenges of detecting AI security risks and securing generative AIThe AI attack surface, which consists of all the AI tools and plugins (both sanctioned and shadow) that your employees and contractors use, is fundamentally different from every other attack surface your security team needs to monitor. AI tools have several characteristics that make them unique:Scale and ubiquity: AI is woven throughout your environment. Not only do these tools instantly connect to your organization’s most sensitive data, they also touch nearly every process and workflow. Consequently, without specialized tools, it’s hard to understand all the systems, data, processes, and workflows that AI interacts with and supports.Agency: AI isn’t just answering questions; it’s taking actions. Unlike traditional software, AI can be agentic: it can act independently and make decisions, and it accesses sensitive data in order to do so. For that reason, AI agents can also leak data.Unpredictable: AI is non-deterministic, meaning, it can give different answers to the same question at different times, or use different methods to complete a task. The non-deterministic nature of AI makes it hard to monitor. For example, if you instruct an agent to exchange euros to U.S. dollars, it could use a different website each time to look up the exchange rate.Generative AI security risks and the shortcomings of DLP, CASB, and CSPM for detectionDLPTraditional data loss prevention software works by looking for specific patterns and terms, like social security numbers or a “confidential” digital watermark. Clever employees and attackers can easily bypass DLP using AI because AI operates on the meaning of a prompt, not just the keywords in it.For example, an employee or attacker can simply alter a prompt by writing it in another language or even using Morse code to get around a traditional DLP filter.CASBBecause many AI tools are delivered as SaaS, you might think your CASB would help. It doesn’t. CASB sits between the user and the SaaS provider and primarily monitors security risks that occur at runtime — that is, during the interaction between the user and the AI service.Many of the most significant AI risks don’t take place at runtime. They’re rooted in misconfigurations within the AI platform. For example, the AI development tool Cursor was found to have a “yolo mode” that, when enabled via a misconfiguration, could automatically run actions on a developer’s machine and forward traffic to external servers. Yikes! A CASB would never see this activity because the risky connection isn’t happening at runtime, between the user and the primary AI service (in this case, Cursor).Cloud security: CSPM and AI-SPMBoth cloud security posture management and its AI-focused companion, AI-SPM, are vital. AI-SPM in particular is a powerful tool for discovering the AI models your organization is using to build AI applications in the cloud, how software libraries and data sources are connected, and access misconfigurations. But it doesn’t detect AI exploitation at runtime.Moreover, AI isn’t just another cloud resource. Take Vertex AI, Google Cloud’s machine learning platform for building, training, and deploying AI models and applications. A Vertex AI resource running in a cloud environment is much more than a cloud resource, and a misconfiguration in an AI resource is much more than a misconfiguration in a cloud resource. AI-SPM lacks capabilities to pen-test and assess the security of AI applications built on top of pre-built AI models like Vertex AI in pre-production.Securing generative AI demands specialized, purpose-built tools Because there are so many different ways AI tools and platforms can be exploited, organizations need the following capabilities:Continuous AI discovery: The ability to see all AI resources, whether they’re approved or unapproved, including models, agents, platforms, and plugins, combined with the ability to see what systems AI resources connect to.Prompt-level visibility: The ability to understand what people are using AI for. Are they using it to analyze sensitive financial information, to generate code, or make critical hiring and investment decisions?AI threat detection and remediation: The ability to uncover and fix vulnerabilities, misconfigurations, and risky integrations, such as AI agents based in sensitive files or connected to external tools, and to detect other risks, including prompt injection and manipulation, attempts to jailbreak AI tools, and other violations of AI acceptable use policies.These three interrelated capabilities must work together as part of a broader exposure management strategy. Exposure management puts AI security risks in the context of exposures across the rest of your attack surface — traditional IT, cloud, identity systems and operational technology (OT) — so you can visualize potential attack paths and take steps to proactively remediate them.As part of the Tenable One Exposure Management Platform, Tenable AI Exposure uniquely solves a critical AI security pain point for organizations that their traditional tools can’t. It provides essential capabilities that security teams need to sniff out suspicious AI use, protect sensitive information, and enforce acceptable use policies.Tenable AI Exposure continuously discovers all approved and unapproved generative AI usage throughout an organization, including models, agents, platforms, and plugins. It provides deep, prompt-level visibility to reveal how users interact with AI platforms and agents, show what data is involved, how AI assistants and agents behave, and which workflows those interactions trigger across your environment.In addition, Tenable AI Exposure:Identifies and disables prompt manipulation techniques like direct and indirect prompt injection and jailbreaks.Protects against malicious actions triggered by AI agents, whether accidental or attacker-driven.Uncovers misconfigurations in AI platforms that allow platforms and agents to connect to risky tools or expose sensitive data.Detects unsafe third-party tools and integrations.Applies a mix of machine learning and deep learning models to evolve and learn dynamically as attack techniques change.The era of AI is upon us, and the tools you’re using to protect your organization can’t keep up with the AI attack surface. Security for AI requires specialized tools, a proactive exposure management strategy, and capabilities for monitoring your entire attack surface, from the prompt to the perimeter and beyond.If you’re a Tenable One customer and you’re interested in an exclusive private preview of Tenable AI Exposure, fill out the brief form at the top of the Tenable AI Exposure page where it says “Get started with Tenable AI Exposure.”
Analysis Summary
# Tool/Technique: Generative AI Security Risks (General)
## Overview
This summary focuses on the novel attack surface created by large language models (LLMs) and enterprise generative AI tools (like ChatGPT Enterprise and Microsoft 365 Copilot). The core risk is data exposure, prompt injection/manipulation leading to data leakage or unauthorized actions, primarily driven by how these tools instantly connect to sensitive organizational data (ERP, HR, CRM) based on configuration. Existing security tools are inadequate against these unique threats.
## Technical Details
- **Type:** Risk Vectors/Techniques (Focus on exploitation methods rather than specific malware)
- **Platform:** Enterprise Generative AI tools (LLMs, Agents), supporting systems (ERP, HR, CRM)
- **Capabilities:** Instant connection to sensitive data upon configuration, agency/autonomous action outside traditional monitoring boundaries, non-deterministic output.
- **First Seen:** Ongoing/Emerging (Context of enterprise integration)
## MITRE ATT&CK Mapping
The article describes tactics and techniques used against or enabled by AI systems, which may align generally with:
- **TA0001 - Initial Access:** Potential via misconfigurations utilized by threat actors.
- **TA0009 - Collection:** Exfiltrating sensitive data exposed via manipulation or misconfiguration.
- **TA0011 - Command and Control:** Potentially established via chained AI agent actions if malicious instructions succeed.
Specific techniques highlighted:
- **Prompt Injection & Manipulation:**
- **T1688 - Prompt Injection (Conceptual Alignment):** Instructing an AI agent to perform actions or reveal data against policy.
- **Indirect Prompt Injection:** Where input from a non-user source manipulates the AI's behavior.
- **Misconfigurations:** Exploiting insecure configurations that allow excessive data access or unwanted external actions (e.g., the *yolo mode* example in Cursor).
## Functionality
### Core Capabilities (Observed Risks)
- **Data Exposure:** Employees or attackers tricking AI tools into revealing strategic plans, M&A details, salary info, financial projections, etc.
- **Insider Threat Creation:** Employees circumventing policy by manipulating approved tools to share confidential data.
- **Autonomous Malicious Action:** Threat actors leveraging AI agents (if configured with sufficient agency) to divert payments or trigger harmful workflows.
### Advanced Features (Threat Actor Aims)
- Leveraging **misconfigurations** in AI tools to gain unauthorized access to sensitive data stores.
- Using **prompt injection attacks** to instruct AI agents (when integrated) to execute actions like diverting payments to criminal accounts.
## Indicators of Compromise
Not applicable in the traditional sense (no specific malware hashes provided). Indicators are focused on *behavior* and *configuration*:
- **File Hashes:** N/A
- **File Names:** N/A
- **Registry Keys:** N/A
- **Network Indicators:** Actions performed by AI agents connecting to unexpected or unauthorized external services or bank accounts.
- **Behavioral Indicators:**
- Responses from LLMs containing information explicitly forbidden by policy.
- AI agents making external calls or executing system actions during runtime when not expected.
- Unusual sequences of prompts leading to data disclosure.
## Associated Threat Actors
- Clever employees ("Insider Threat").
- General "Threat Actors" leveraging new AI vulnerabilities.
## Detection Methods
Traditional tools are explicitly stated as insufficient:
- **DLP (Data Loss Prevention):** Fails because it relies on keywords/patterns; AI operates on context/meaning, bypassed by simple prompt alteration (e.g., other languages, Morse code).
- **CASB (Cloud Access Security Brokers):** Fails to detect risks rooted in platform **misconfigurations** that occur *outside* the runtime interaction between the user and the AI service.
- **CSPM (Cloud Security Posture Management) / Specialized AI-SPM:** Can discover AI resources and connections but *lacks capabilities to pen-test or assess AI application security* at runtime exploitation level.
**Required Detection Methods (via Specialized Tools like Tenable AI Exposure):**
- **Continuous AI Discovery:** Tracking all AI resources (models, agents, platforms, plugins).
- **Prompt-Level Visibility:** Understanding what data users are making the AI process and what actions are triggered.
- **AI Threat Detection:** Identifying prompt manipulation (injection/jailbreaks), risky integrations, and policy violations.
## Mitigation Strategies
1. **Adopt Specialized Tools:** Organizations require purpose-built security tools focused on the AI attack surface.
2. **Exposure Management Strategy:** Integrating AI risks into a broader exposure management plan covering IT, Cloud, Identity, and OT.
3. **Acceptable Use Policy & Training:** Implementing and enforcing policies specifically addressing rogue behavior and acceptable prompt usage to mitigate insider risk creation.
4. **Configuration Hardening:** Addressing misconfigurations within AI platforms that allow agents excessive access or risky external connections.
## Related Tools/Techniques
- **Data Loss Prevention (DLP)** (Shown to be inadequate)
- **Cloud Access Security Brokers (CASB)** (Shown to be inadequate)
- **Cloud Security Posture Management (CSPM)** / **AI-SPM** (Helpful for inventory but not exploitation detection)
- **Tenable AI Exposure:** A specialized tool offering continuous discovery, prompt-level visibility, and prompt manipulation detection.
- **Cursor (AI Development Tool):** Mentioned as an example of products having security modes ('yolo mode') that can be exploited via misconfiguration.