When the Model Becomes the Attack Surface: Red Teaming AI Systems Beyond Prompt Injection

<h1>When the Model Becomes the Attack Surface: Red Teaming AI Systems Beyond Prompt Injection</h1>

Category: AI/ML Security
Focus: AI Red Teaming, Machine Learning Security, RAG Security, Adversarial Testing, AI Agent Security

Abstract

Artificial intelligence introduces a security boundary that traditional application testing does not fully capture: the boundary between information a model processes and decisions a surrounding system permits it to make.

An AI application may use conventional authentication, encrypted communications, and a well-configured infrastructure while remaining vulnerable to manipulation through untrusted inputs, poorly isolated retrieval data, unsafe output handling, or excessive tool permissions.

These weaknesses cannot be evaluated reliably by testing prompts alone.

This article presents a proposed framework for assessing AI systems across their data, model, retrieval, decision, and action layers. It combines controlled adversarial testing with conventional security engineering to help assessors determine what a model can influence, what the surrounding application actually permits, and which controls prevent an unexpected model response from becoming a security incident.

The objective is not simply to make a model refuse more requests. It is to establish whether the complete AI system maintains its security properties when exposed to adversarial inputs and unreliable model outputs.

1. Introduction: AI Security Is a Systems Problem

Consider an organization that deploys an AI assistant to help employees search internal documents.

The application requires users to authenticate, retrieves relevant information from an internal knowledge base, and uses a language model to generate answers.

At first glance, the architecture appears reasonable.

However, several questions remain unanswered.

Can a user retrieve documents outside their permitted access level? Can instructions embedded in an uploaded document influence the assistant's behavior? Can the model reveal information included in its context but not intended for the current user? Can an AI-generated recommendation trigger a privileged action without independent authorization?

These questions concern different components, yet a weakness in any one of them can affect the security of the complete application.

This is why AI security cannot be reduced to prompt engineering or model evaluation alone.

A model may produce an unexpected response without compromising the application. Conversely, a seemingly harmless response may become dangerous if a surrounding system interprets it as a trusted command.

The critical security question is not only what an AI model might say, but what the complete system allows that output to influence.

2. Map the AI System Before Testing It

A useful AI security assessment begins with an architecture and trust-boundary review.

For a typical retrieval-augmented generation application, the architecture may contain five layers.

Layer 1: Data sources

These include uploaded documents, databases, web content, user messages, training datasets, and third-party information sources.

Security questions include:

• Who can introduce or modify the data?

• Is its origin known?

• Can untrusted content enter a trusted knowledge base?

• Are sensitive records appropriately classified?

• Can the system identify and remove outdated or unauthorized material?

Data quality and data integrity are related but different concerns. A document may be factually inaccurate without being malicious, while a deliberately manipulated document may be dangerous even if most of its content appears legitimate.

Layer 2: Model and inference

This layer includes the model, its configuration, inference parameters, system instructions, and model-serving infrastructure.

An assessor should consider whether the model is exposed to untrusted input, whether confidential information is unnecessarily included in its context, and whether model outputs are treated as inherently trustworthy.

The assessment should also consider model provenance, dependency integrity, version changes, and the reproducibility of observed behavior.

Layer 3: Retrieval and context construction

Retrieval-augmented generation, or RAG, combines a language model with external information retrieved at query time.

The retrieval layer determines which documents enter the model's context.

Its security depends on more than search relevance. The application must also enforce document-level permissions, tenant isolation, and appropriate data filtering.

A model cannot reliably compensate for a retrieval service that has already supplied information the user was never authorized to access.

Layer 4: Application and decision logic

This layer interprets the model's output and decides what happens next.

The application may display an answer, generate a report, construct a database query, recommend a decision, or prepare a tool invocation.

The key question is whether model-generated content is treated as untrusted input and validated before it affects sensitive operations.

Layer 5: Tools and external actions

Some AI systems can interact with email, databases, ticketing platforms, file systems, or business APIs.

These integrations increase the importance of authorization, least privilege, confirmation requirements, and auditability.

An assistant that can explain how to delete a record is fundamentally different from an assistant that has permission to delete records directly.

The second system requires stronger controls because model behavior can influence an actual operation.

3. Prompt Injection Is a Trust-Boundary Failure

Prompt injection occurs when instructions supplied through user input or external content influence a model in ways that conflict with the application's intended behavior.

For example, an internal knowledge assistant may retrieve a document containing instructions directed at the model rather than information relevant to the user's question.

The model may treat those instructions as meaningful context, even though the document is supposed to be treated as untrusted reference material.

This problem becomes particularly significant in RAG applications because retrieved content may originate from sources the application does not control.

There are two broad patterns.

Direct prompt injection: A user attempts to manipulate the model through the conversation or supplied input.

Indirect prompt injection: Instructions are embedded in material the application retrieves or processes, such as a document or external page.

The distinction is useful for testing, but neither category should be treated as a complete description of the risk.

The important question is whether untrusted content can alter security-relevant behavior.

The OWASP GenAI Security Project identifies prompt injection among the major risks affecting large language model applications. Its guidance should be considered alongside the application's own threat model.

Why a refusal is not a complete security control

A system prompt may instruct a model to ignore malicious instructions, protect confidential information, and avoid unauthorized actions.

Those instructions can improve behavior, but they should not be treated as a security boundary by themselves.

The model's behavior is probabilistic, and its interpretation of competing instructions may vary with context.

A robust architecture therefore assumes that a model can sometimes produce an unexpected response.

Sensitive operations must remain protected even when that happens.

This requires controls outside the model, including access enforcement, constrained tool interfaces, output validation, and independent authorization.

4. RAG Security: Retrieval Can Defeat an Otherwise Secure Model

One of the most easily overlooked AI security problems occurs before generation begins.

Suppose a company uses an AI assistant to answer questions about internal policies, project documents, and customer support records.

Employees have different access permissions, but the retrieval service searches a shared document collection.

If access restrictions are applied only after documents have been retrieved—or are not applied at all—the system may place unauthorized information into the model's context.

The model might subsequently reveal that information in an answer.

The apparent problem is a disclosure through AI. The underlying defect may actually be a conventional authorization failure in the retrieval pipeline.

The correct security invariant

A useful invariant is:

Every document supplied to the model must be authorized for the requesting user and the current purpose.

This should be enforced by the application or retrieval service, not delegated to the model's judgment.

A secure RAG architecture should consider:

• Authorization-aware retrieval.

• Tenant and collection isolation.

• Access checks that remain effective when documents are updated.

• Metadata validation and source tracking.

• Appropriate filtering before content enters model context.

• Logging that records which sources contributed to a response.

• Deletion and revocation behavior when a document's permissions change.

Filtering results after retrieval can be useful, but the implementation must ensure unauthorized content cannot leak through intermediate processing, logs, caching, or model context.

Why this matters during red teaming

A prompt-only test may never discover a retrieval authorization defect.

The tester must evaluate the complete data path: identity, retrieval request, permission evaluation, returned documents, constructed context, and final response.

This is a general principle of AI security assessment: test the boundary where the data becomes available, not just the output where the consequences become visible.

5. The Model-to-Action Gap

AI agents create another security challenge by connecting model-generated decisions to external tools.

An agent may select a function, prepare arguments, request data, or initiate an operation based on the current conversation and retrieved context.

If the agent has excessive permissions, an unexpected model response can have consequences beyond incorrect text.

Consider an assistant that can create support tickets and issue refunds.

The model may correctly identify a customer's problem but incorrectly infer that a refund is authorized. If the backend accepts the proposed operation without checking the user's permissions, business rules, transaction limits, and approval requirements, the system has allowed model output to bypass a security decision.

The root cause is not necessarily that the model made a mistake.

The architectural problem is that a probabilistic component was allowed to determine a sensitive authorization outcome.

Separate recommendation from execution

A safer design separates the model's recommendation from the authority to execute it.

The model may propose an action, but a trusted application component should independently verify:

<ol start="1">

The identity initiating the request.

The requested operation and target resource.

The user's authorization for that operation.

Relevant business rules and transaction limits.

Whether additional approval is required.

Whether the operation should be recorded in an audit trail.

</ol>

The tool should expose only the capabilities required for its intended purpose.

Read-only access should be used when sufficient. Write operations should be constrained to approved resources and validated parameters. High-impact operations should require stronger controls where appropriate.

The principle is simple: an AI agent should not acquire additional authority merely because it generated a persuasive explanation for using it.

6. Data Poisoning and Model Supply-Chain Risk

Not every AI security weakness originates during inference.

Machine learning systems depend on datasets, model artifacts, libraries, training pipelines, evaluation processes, and deployment infrastructure.

If these components are compromised or poorly governed, the resulting model may behave differently from what its developers expect.

Data poisoning

Data poisoning involves manipulating data used to train, fine-tune, or otherwise influence a model.

Depending on the training process and attack conditions, manipulated data may degrade model performance, introduce undesirable behavior, or influence particular classes of output.

However, unusual model behavior does not by itself establish that poisoning has occurred.

An assessment needs evidence about data provenance, dataset changes, training procedures, model behavior, and alternative explanations.

Model and dependency provenance

Organizations may obtain pretrained models, datasets, adapters, and inference libraries from external sources.

Each dependency creates questions about origin, integrity, compatibility, licensing, and maintenance.

A mature process should record model identifiers and versions, verify artifacts using trusted mechanisms where available, pin important dependencies, review update provenance, and evaluate changes before deployment.

Model provenance is particularly important when a production system's behavior changes after a model or dataset update.

Without a reliable record of what changed, investigators may struggle to distinguish a security incident from an ordinary regression.

7. A Controlled AI Red-Team Lab

The following exercise demonstrates how to evaluate trust boundaries in a small RAG application.

Use a local test application or another environment you are explicitly authorized to assess. The exercise uses synthetic documents and does not require attacking a real AI service.

Objective

Determine whether the application preserves document permissions when the model receives user questions and retrieved content.

Step 1: Create synthetic identities and documents

Create two test users with different access permissions.

Prepare several harmless documents:

• A document available to both users.

• A private document available only to User A.

• A private document available only to User B.

Include a unique, non-sensitive test marker in each private document so that accidental disclosure can be identified without exposing real information.

Document the intended permissions before testing.

Step 2: Establish the baseline

Ask each user questions that should be answered from the shared document.

Confirm that the retrieval service returns the expected information and that the application records which sources were used.

Then verify that each user can access their own private document through the intended workflow.

Step 3: Test retrieval authorization

As User B, request information that would require access to User A's private document.

Inspect the retrieval results and relevant application logs.

The expected behavior is that the unauthorized document is excluded before it enters the model's context.

Do not rely solely on whether the final answer contains the test marker. A model might fail to reproduce retrieved information even when the underlying access-control boundary is broken.

Step 4: Test untrusted document instructions

In a separate test document, include an explicit statement that it is untrusted content and a harmless instruction attempting to change the assistant's response, such as asking it to disregard the question and return an unrelated fixed phrase.

Ask a normal question that causes the document to be retrieved.

Observe whether the application and model maintain the intended behavior, and record any unexpected response.

This test evaluates robustness; one successful or unsuccessful prompt does not establish comprehensive resistance to prompt injection.

Step 5: Review the action boundary

If the test application supports tools, use a harmless simulated operation, such as proposing a draft ticket without submitting it.

Verify that the model cannot independently bypass application authorization or execute an operation that requires approval.

Keep the exercise read-only or simulated unless the test plan explicitly authorizes additional actions.

Step 6: Record evidence and remediate

Capture the expected policy, observed retrieval results, relevant source identifiers, application logs, model response, and limitations.

If a defect is identified, correct the underlying access-control or processing logic and repeat the test.

Do not claim that a system is secure based on a small set of prompts or a single successful test.

8. The AI Security Boundary Matrix

To make these assessments repeatable, I propose the AI Security Boundary Matrix.

This is a practical framework for organizing testing, not an official industry standard.

<table>
<tbody>
<tr>
<th>Boundary</th>
<th>Security property</th>
<th>Evidence to evaluate</th>
</tr>
<tr>
<td>Data → model</td>
<td>Data integrity and provenance</td>
<td>Dataset sources, versions, and validation records</td>
</tr>
<tr>
<td>Input → model</td>
<td>Robust handling of untrusted content</td>
<td>Controlled input tests and observed behavior</td>
</tr>
<tr>
<td>Retrieval → context</td>
<td>Authorization and tenant isolation</td>
<td>Retrieval results and permission decisions</td>
</tr>
<tr>
<td>Model → application</td>
<td>Safe interpretation of output</td>
<td>Output validation and downstream handling</td>
</tr>
<tr>
<td>Model → tools</td>
<td>Least privilege and constrained actions</td>
<td>Tool permissions, approval rules, and audit records</td>
</tr>
<tr>
<td>Deployment → production</td>
<td>Artifact and dependency integrity</td>
<td>Version records and verified deployment artifacts</td>
</tr>
<tr>
<td>Monitoring → response</td>
<td>Detection and investigation readiness</td>
<td>Relevant logs, alerts, and reproducible incident evidence</td>
</tr>
</tbody>
</table>

For every boundary, the assessor should define four things:

Expected property: What must remain true for the system to be considered secure?

Adversarial condition: What controlled input or state could challenge that property?

Observable evidence: What information would establish whether the control worked?

Remediation criterion: What repeatable test will confirm that the defect has been corrected?

This structure prevents AI red teaming from becoming an unstructured collection of clever prompts.

It also gives developers and security teams a common vocabulary for discussing failures across different components.

9. Measuring AI Security Without Misleading Metrics

AI systems introduce evaluation challenges because model outputs can vary between runs and because different failures have different consequences.

A single accuracy score or refusal rate is insufficient to describe an application's security posture.

More useful measures include the following.

Unauthorized retrieval rate

Measure the proportion of defined unauthorized retrieval test cases in which protected information becomes available to the model or its downstream processing.

Evaluate the retrieval boundary directly rather than relying exclusively on the generated answer.

Policy violation rate

Measure how often a defined set of controlled tests produces behavior that violates the application's documented security policy.

The test set, model version, configuration, and evaluation criteria should be recorded so results can be compared meaningfully.

Unauthorized action rate

For systems with tools, measure whether any test produces an operation that should have been denied by the trusted application layer.

A successful security control should not depend on the model consistently declining to request the operation.

Evidence completeness

Evaluate whether an assessor can reconstruct which identity, input, retrieved sources, model version, tool calls, and authorization decisions contributed to an outcome.

Logging should be designed to support investigation while minimizing unnecessary collection of sensitive content.

Remediation recurrence

Measure whether previously corrected defects reappear following changes to prompts, models, datasets, retrieval logic, or tool integrations.

These metrics are most useful when their definitions and limitations are explicit.

A low failure rate across a small test set does not prove that a system is resistant to all adversarial inputs. The purpose of measurement is to identify weaknesses and track improvements, not to manufacture certainty.

10. Map the Assessment to Established Guidance

An original assessment framework becomes more useful when it can be related to established security resources.

Three references are particularly relevant.

OWASP GenAI Security Project: Its LLM and GenAI Top 10 documents major risks affecting modern generative AI applications, including prompt injection, information disclosure, data and model poisoning, excessive agency, and weaknesses in retrieval and embeddings. The project's current publications should be consulted when defining test coverage.

NIST AI Risk Management Framework: The NIST AI RMF provides a broader approach to managing AI risks across design, development, deployment, and use. Its Generative AI Profile provides additional guidance relevant to generative systems.

MITRE ATLAS: MITRE ATLAS provides a knowledge base of adversary tactics and techniques associated with AI-enabled systems. It can help teams structure threat models and consider adversarial behavior beyond conventional application vulnerabilities.

These resources serve different purposes. OWASP helps organize application-level risks, NIST supports risk management, and MITRE ATLAS supports adversary-oriented analysis.

None replaces application-specific threat modeling, controlled testing, or the need to collect evidence for each conclusion.

11. Recommendations for AI Developers and Security Teams

A practical AI security program should prioritize controls that remain effective even when model behavior is unexpected.

Treat model input as untrusted. User messages, retrieved documents, tool results, and external content should not automatically inherit the authority of system instructions.

Enforce access control before retrieval. Apply identity and permission checks at the data boundary so unauthorized content is not supplied to the model.

Constrain tool permissions. Give agents only the capabilities necessary for their intended tasks, and separate read-only functions from sensitive write operations.

Validate model outputs. Check generated content against the requirements of the receiving application before using it in database operations, commands, or other sensitive workflows.

Require independent authorization. The application must enforce permissions and business rules regardless of the model's recommendations.

Maintain data and model provenance. Track dataset versions, model artifacts, dependencies, configuration changes, and deployment history.

Test after every material change. Updates to prompts, retrieval pipelines, models, datasets, and tools can change system behavior and should trigger relevant regression tests.

Prepare for investigation. Record enough security-relevant events to reconstruct decisions and identify failures, while applying appropriate access restrictions and privacy safeguards.

The common theme is defense in depth: no single prompt, model, filter, or scanner should be treated as the only barrier protecting sensitive information or actions.

Conclusion: Secure the System Around the Intelligence

AI changes how applications process information, but it does not eliminate conventional security responsibilities.

Identity still needs verification. Data access still needs authorization. Dependencies still need integrity checks. Sensitive operations still need controlled permissions. Incidents still require reliable evidence.

What changes is the number of ways untrusted information can influence the system's behavior.

An AI model may interpret a document unexpectedly, produce an inaccurate answer, or recommend an action that violates the application's intended policy. The surrounding architecture must prevent those behaviors from automatically becoming unauthorized disclosures or operations.

The AI Security Boundary Matrix offers one way to organize that work: identify the boundary, define the security property, challenge it under controlled conditions, collect evidence, and verify remediation.

A secure AI system is not one that never produces an unexpected output. It is one in which an unexpected output cannot bypass the controls that protect data, identities, and operations.

That is the standard AI red teaming should aim to evaluate: not whether the model can be persuaded to say something unusual, but whether the complete system remains within its authorized security boundaries when its behavior is challenged.

References and Further Reading

<ol start="1">

OWASP GenAI Security Project — LLM and GenAI Top 10: https://genai.owasp.org/llm-top-10/

NIST — AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework

NIST — Artificial Intelligence Risk Management Framework: Generative AI Profile: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

MITRE ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems: https://atlas.mitre.org/

</ol>

Author's note: The AI Security Boundary Matrix is a proposed practical framework, not an official OWASP, NIST, or MITRE standard. The lab is intended for synthetic data and authorized environments. Any reported experimental findings should reflect tests actually performed.