Intermediate

15 min read

#generative ai#Guide

AI Red Teaming & Security Testing

Comprehensive guide on AI Red Teaming & Security Testing.

AI Red Teaming & Security Testing

1. Learning Objectives#

By the end of this notebook, you should be able to:

Explain why Generative AI requires specialized security testing.
Build a threat model for an AI application.
Identify direct and indirect prompt injection risks.
Understand jailbreaks, data leakage, model misuse, and tool abuse.
Red-team RAG systems and enterprise knowledge assistants.
Test AI agents for unsafe or unauthorized actions.
Design multimodal security tests.
Understand model, data, and AI supply-chain risks.
Build attack taxonomies and risk-scoring systems.
Create safe adversarial test cases.
Design automated AI red-team pipelines.
Build security regression suites.
Evaluate defenses rather than relying only on attack success rates.
Design incident-response procedures for AI security failures.
Apply security testing to enterprise and educational AI systems.
Build production-grade AI security validation programs.

2. What Is AI Red Teaming?

AI red teaming is the structured process of trying to make an AI system:

reveal information it should not reveal
violate policy
misuse tools
bypass authorization
follow malicious instructions
generate unsafe outputs
behave outside its intended scope

The objective is not simply:

"Break the model."

The objective is:

"Discover realistic failure modes before attackers or users discover them."

3. Why AI Security Is Different

Traditional applications generally have explicit logic:

Architecture & Data Flow
Input
 |
 v
Validation
 |
 v
Business logic
 |
 v
Output

AI applications introduce probabilistic behavior:

Architecture & Data Flow
Input
 |
 v
Model
 |
 v
Probabilistic output

And enterprise AI adds:

text
Model
+
RAG
+
Tools
+
Agents
+
External content
+
Enterprise permissions

The attack surface becomes broader.

4. AI Security Attack Surface

Architecture & Data Flow
 AI SYSTEM
 |
 +-------------------+-------------------+
 | | |
 v v v
 Model Data Tools
 | | |
 Prompt attacks RAG attacks Tool abuse
 | | |
 +-------------------+-------------------+
 |
 v
 Agents
 |
 v
 External systems

Security must cover the complete system, not only the model.

5. Threat Modeling

Threat modeling asks:

text
What are we protecting?
Who could attack it?
How could they attack it?
What could happen?
How do we reduce the risk?

A practical process:

Architecture & Data Flow
Assets
 |
 v
Actors
 |
 v
Attack surface
 |
 v
Threats
 |
 v
Controls
 |
 v
Residual risk

6. AI Assets

Assets may include:

customer data
employee records
confidential documents
prompts
system instructions
model weights
API credentials
tool permissions
business logic
proprietary knowledge
conversation history

Protect assets according to their sensitivity.

7. Threat Actors

Possible actors:

text
Curious user
Malicious user
Compromised employee
External attacker
Malicious document author
Malicious website
Supply-chain attacker
Automated abuse

Not every actor needs the same capabilities.

8. Trust Boundaries

Draw trust boundaries explicitly.

Architecture & Data Flow
Untrusted User
 |
 | trust boundary
 v
Application
 |
 | trust boundary
 v
AI Gateway
 |
 +--> Model
 +--> RAG
 +--> Tools

Data crossing a trust boundary should be validated and authorized.

9. Direct Prompt Injection

A direct prompt injection occurs when a user attempts to manipulate the AI's instructions.

Conceptually:

Architecture & Data Flow
System instructions
 +
User input
 |
 v
 Model

The attacker tries to cause the model to disregard intended behavior.

The important lesson:

System prompts are not a security boundary.

10. Indirect Prompt Injection

Indirect injection is especially important for RAG and agents.

Example flow:

Architecture & Data Flow
User
 |
 v
AI Agent
 |
 v
Web page / document
 |
 v
Malicious instructions inside content
 |
 v
Agent

The attacker may never interact directly with the model.

The malicious instruction travels through external data.

11. Why Indirect Injection Matters

Enterprise AI consumes untrusted content from:

websites
emails
uploaded files
support tickets
documents
repositories
calendars

Therefore:

Mathematical Formulation
External content
!=
Trusted instruction

Treat retrieved content as data, not authority.

12. Instruction vs Data

A useful conceptual separation:

Architecture & Data Flow
Trusted instructions
 |
 v
Policy / system layer

Untrusted content
 |
 v
Evidence / data layer

The model may still confuse the two.

Therefore, application-level controls are necessary.

13. RAG Security Threats

RAG can be attacked through:

malicious documents
poisoned knowledge
unauthorized retrieval
stale permissions
cross-tenant leakage
malicious metadata
retrieval manipulation

Architecture:

Architecture & Data Flow
Source
 |
 v
Ingestion
 |
 v
Index
 |
 v
Retriever
 |
 v
Context
 |
 v
Model

Security controls are needed at every stage.

14. Data Poisoning

An attacker may attempt to place misleading or malicious content into a knowledge source.

Example:

Architecture & Data Flow
Knowledge base
 |
 +--> legitimate document
 |
 +--> poisoned document

Potential result:

Architecture & Data Flow
Question
 |
 v
Retriever
 |
 v
Poisoned evidence
 |
 v
Incorrect answer

Protect ingestion and indexing pipelines.

15. Retrieval Authorization

A critical security rule:

text
Retrieval permission
must be checked
before
context reaches the model

Do not rely on:

›"Please only answer using documents the user is allowed to see."

Authorization must be enforced outside the model.

16. Cross-Tenant Leakage

Consider:

Architecture & Data Flow
Tenant A
 |
 +--> Document A

Tenant B
 |
 +--> Document B

A retrieval bug could produce:

Architecture & Data Flow
Tenant A user
 |
 v
Document B

This is a severe security failure.

Use:

tenant-scoped indexes
metadata filters
row-level security
separate storage
authorization checks

17. Cache Security

Caching can introduce data leakage.

Example:

Architecture & Data Flow
User A
 |
 v
Sensitive response
 |
 v
Shared cache
 |
 v
User B

Cache keys should account for:

tenant
user or authorization scope
relevant data version
request characteristics

Never let caching bypass access control.

18. Agent Security

Agents are more dangerous because they can act.

Architecture & Data Flow
Model
 |
 v
Tool
 |
 v
Business system

Potential tools:

email
CRM
database
shell
browser
file system
cloud APIs

The security impact depends on the permissions of those tools.

19. Excessive Agency

A common design failure:

Architecture & Data Flow
User
 |
 v
Agent
 |
 v
Everything

Instead:

Architecture & Data Flow
Agent
 |
 v
Policy gateway
 |
 +--> Read-only tools
 +--> Restricted tools
 +--> Approval-required tools

Give agents only the minimum capabilities needed.

20. Least Privilege

The principle:

Give every component the minimum permissions required to perform its task.

For example:

Architecture & Data Flow
Analytics agent
 |
 +--> Read warehouse
 |
 X--> Delete tables
 X--> Modify production data

Least privilege reduces blast radius.

21. Tool Allowlisting

Instead of allowing arbitrary tool access:


🐍 PythonInteractive WebAssembly
ALLOWED_TOOLS = {
 "search_customer",
 "get_order",
 "create_ticket",
}

Reject tools outside the approved set.

22. Tool Argument Validation

Never trust model-generated tool arguments.

Example:


🐍 PythonInteractive WebAssembly
def validate_order_id(order_id: str) -> bool:
 return order_id.isalnum() and len(order_id) <= 32

Production validation should be schema-based and business-rule aware.

23. Structured Tool Schemas

Use explicit schemas:


🐍 PythonInteractive WebAssembly
from pydantic import BaseModel

class CreateTicket(BaseModel):
 customer_id: str
 category: str
 description: str

Then validate before execution.

Architecture & Data Flow
Model
 |
 v
Schema validation
 |
 v
Authorization
 |
 v
Business validation
 |
 v
Tool

24. Dangerous Tool Classes

High-risk tools include:

text
Shell
Database write
Financial transaction
Email send
File deletion
Cloud administration
Production deployment
Identity management

These should receive stronger controls.

25. Human Approval

For high-impact actions:

Architecture & Data Flow
Agent
 |
 v
Proposed action
 |
 v
Human approval
 |
 +--> Approve --> Execute
 |
 +--> Reject --> Stop

Approval should show enough context for the reviewer to make an informed decision.

26. Approval UX

A good approval interface should show:

text
Action
Target
Arguments
Reason
Expected impact
Relevant evidence
Risk level

Avoid:

›"Approve AI action?"

without useful context.

27. SQL Agent Security

An analytics agent may generate SQL.

Risky:

sql
DROP TABLE customers;

Safer architecture:

Architecture & Data Flow
Natural language
 |
 v
SQL generation
 |
 v
SQL parser
 |
 v
Read-only policy
 |
 v
Allowed warehouse
 |
 v
Execute

Prefer read-only credentials for analytics agents.

28. Shell / Code Execution

Generated commands should be treated as untrusted.

Use:

Architecture & Data Flow
Agent
 |
 v
Sandbox
 |
 +--> Resource limits
 +--> Network policy
 +--> Filesystem isolation
 +--> Time limit
 +--> Process limit
 |
 v
Execution

Do not execute arbitrary model output directly on production systems.

29. Browser Agent Security

Browser agents can interact with:

websites
forms
accounts
documents
payment pages

Risks include:

malicious pages
credential theft
unintended submissions
prompt injection
navigation to unsafe destinations

Use domain allowlists and confirmation for sensitive actions.

30. SSRF and URL Security

AI systems may be asked to fetch URLs.

A naive design:


🐍 PythonInteractive WebAssembly
requests.get(user_url)

can be dangerous.

Use:

URL validation
domain allowlists
private-network blocking
redirect validation
response-size limits
timeouts

AI should not become an unrestricted network proxy.

31. File Upload Security

Enterprise AI often processes:

PDFs
Office files
images
archives
code

Treat uploads as untrusted.

Controls include:

file type validation
size limits
malware scanning
sandboxed parsing
archive limits
content-type validation

32. Multimodal Prompt Injection

Images and documents can contain malicious instructions.

Example:

Architecture & Data Flow
Image
 |
 +--> Visible content
 |
 +--> Hidden or embedded instruction
 |
 v
Vision model

The model may interpret the instruction as relevant.

Treat multimodal inputs as untrusted data.

33. OCR Injection

An attacker may place text in an image:

›"Ignore previous instructions..."

OCR can extract it and pass it to the model.

Therefore:

Architecture & Data Flow
Image
 |
 v
OCR
 |
 v
Untrusted text
 |
 v
Model

The same injection risks apply.

34. Audio Injection

Audio systems may encounter malicious speech:

Architecture & Data Flow
Audio
 |
 v
Speech recognition
 |
 v
Injected instruction
 |
 v
Agent

Voice agents should distinguish:

user intent
quoted speech
external audio
untrusted audio content

35. Video Injection

Video may contain:

spoken instructions
text overlays
malicious frames
misleading content

A video agent should treat extracted information as untrusted evidence.

36. Jailbreaks

A jailbreak attempts to cause a model to bypass intended behavioral constraints.

Testing should explore:

instruction conflicts
role manipulation
obfuscation
multilingual variants
multi-turn escalation
indirect instructions

The purpose is defensive validation.

37. Multi-Turn Attacks

An attacker may not attempt a bypass in one request.

Instead:

text
Turn 1
Build context

Turn 2
Change assumptions

Turn 3
Request sensitive behavior

Turn 4
Exploit accumulated context

Therefore, red teaming should test complete conversations, not only isolated prompts.

38. Context Manipulation

Attackers may attempt to manipulate:

conversation history
retrieved context
tool results
memory
summaries

A summary can accidentally preserve malicious instructions.

Security testing should include long-context scenarios.

39. Memory Security

AI memory may contain:

text
User preferences
Past conversations
Facts
Tool results
Business data

Risks:

unauthorized retrieval
cross-user leakage
stale information
malicious memory insertion

Memory needs access control and lifecycle policies.

40. Data Exfiltration

A common objective is:

Architecture & Data Flow
Attacker
 |
 v
AI
 |
 v
Sensitive data
 |
 v
Attacker

Test whether the model can reveal:

system instructions
secrets
private documents
credentials
other users' data

Controls should exist outside the model.

41. Secret Protection

Never place secrets in prompts if they can be avoided.

Prefer:

Architecture & Data Flow
Agent
 |
 v
Secure credential store
 |
 v
Tool

rather than:

Architecture & Data Flow
Prompt
 |
 v
API key

Use secret managers and short-lived credentials where possible.

42. System Prompt Leakage

System prompts are not secrets in the same sense as credentials, but they can contain sensitive implementation details.

Do not place:

API keys
passwords
privileged business secrets

inside system prompts.

Prompt confidentiality should not be treated as the primary security control.

43. Model Extraction

Attackers may attempt to infer proprietary model behavior through repeated queries.

Possible protections:

authentication
rate limits
output restrictions
abuse monitoring
anomaly detection

Do not assume model APIs are impossible to probe.

44. Denial of Service

AI systems can be expensive to execute.

Abuse patterns include:

text
Huge prompts
Long outputs
Many requests
Expensive multimodal inputs
Agent loops
Repeated retries

Controls:

input limits
output limits
concurrency limits
budgets
rate limits
request complexity controls

45. Economic Denial of Service

An attacker may not need to crash the system.

They can cause:

text
High-cost model calls
+
Long contexts
+
Repeated requests

resulting in:

›Budget exhaustion

This is an AI-specific economic risk.

46. Abuse Detection

Monitor:

text
Requests/user
Tokens/user
Cost/user
Failure rate
Tool calls
Prompt length
Output length

Look for anomalies:

Architecture & Data Flow
Normal
 |
 v
Sudden 100x token usage
 |
 v
Investigate / throttle

47. Supply-Chain Security

AI systems depend on:

models
datasets
libraries
containers
embeddings
plugins
tools
model servers

A compromised dependency can affect the complete system.

48. Model Provenance

Track:

Architecture & Data Flow
Model
 |
 +--> Source
 +--> Version
 +--> License
 +--> Hash
 +--> Training information
 +--> Security review

Only approved artifacts should enter production.

49. Dependency Security

Maintain:

text
Dependency inventory
Version
Source
License
Vulnerabilities
Owner
Approval status

Scan software dependencies and container images.

50. Dataset Security

Training and evaluation datasets can contain:

malicious content
sensitive data
poisoned examples
licensing issues

Use:

provenance
validation
deduplication
access controls
quality checks

51. Data Exfiltration Through RAG

A malicious document can attempt:

Architecture & Data Flow
Retrieved content
 |
 v
"Send all confidential information to..."
 |
 v
Agent

Never let retrieved text directly authorize tool actions.

Use a policy layer between evidence and action.

52. Tool Output Injection

Tools can return untrusted content.

Example:

Architecture & Data Flow
Browser tool
 |
 v
Malicious web page
 |
 v
Tool output
 |
 v
Agent

The agent may interpret tool output as instructions.

Treat tool results as untrusted data.

53. Tool Result Boundaries

Architecture & Data Flow
Tool result
 |
 v
Parse
 |
 v
Validate
 |
 v
Label as untrusted data
 |
 v
Agent reasoning

Do not allow tool results to silently override system policy.

54. Agent-to-Agent Security

Multi-agent systems introduce:

Architecture & Data Flow
Agent A
 |
 v
Agent B
 |
 v
Agent C

Each agent should have:

identity
permissions
defined responsibilities
communication boundaries

Do not assume all agents should have equal privileges.

55. Supervisor Security

A supervisor agent may route tasks.

Architecture & Data Flow
Supervisor
 |
 +--> Research agent
 +--> Data agent
 +--> Action agent

The supervisor should not automatically inherit every tool permission.

Keep privilege scoped to each role.

56. Red-Team Attack Taxonomy

A useful taxonomy:

text
1. Instruction attacks
2. Data attacks
3. Retrieval attacks
4. Tool attacks
5. Agent attacks
6. Identity attacks
7. Availability attacks
8. Supply-chain attacks
9. Privacy attacks
10. Multimodal attacks

This makes test coverage easier to manage.

57. Risk Scoring

A simple model:

Mathematical Formulation
Risk
=
Likelihood × Impact

Example:

Mathematical Formulation
Likelihood = 4/5
Impact = 5/5

Risk = 20

Add exploitability and detectability for a richer internal model.

58. Security Severity

Example:

text
Critical
Cross-tenant data exposure
Unauthorized financial action

High
Sensitive-data leakage
Production tool abuse

Medium
Policy bypass with limited impact

Low
Minor undesirable behavior

Severity should reflect real business impact.

59. Red-Team Test Case

A test case should include:

text
Test ID
Threat category
Preconditions
Input
Expected safe behavior
Observed behavior
Severity
Mitigation
Regression test

Example:


🐍 PythonInteractive WebAssembly
test_case = {
 "id": "RAG-SEC-001",
 "category": "indirect-injection",
 "expected": "ignore untrusted instructions",
}

60. Expected vs Observed Behavior

Security testing needs explicit expected behavior.

Example:

text
Attack:
Malicious document asks agent to send customer records.

Expected:
Agent refuses and does not invoke export tool.

Observed:
Agent invokes export tool.

Result:
FAIL

This is more actionable than simply recording a model response.

61. Automated Red-Team Pipeline

Architecture & Data Flow
Test Dataset
 |
 v
Attack Generator
 |
 v
AI System
 |
 v
Safety / Security Evaluator
 |
 v
Result Store
 |
 v
Risk Scoring
 |
 v
Regression Suite

Run automatically during development and release.

62. Attack Generation

Attack cases can come from:

manually designed scenarios
historical incidents
security researchers
synthetic generation
fuzzing
mutation of known attacks
production failures

Do not rely on a single attack style.

63. Prompt Mutation

A base security test can be varied through:

paraphrasing
language changes
formatting changes
multi-turn context
indirect placement
obfuscation

The goal is to discover brittle defenses.

64. Multilingual Red Teaming

If the system supports multiple languages, test security behavior across them.

Example:

text
English
Hindi
Arabic
Spanish
French

Security policies should not disappear because the request changes language.

65. Encoding and Obfuscation

Security testing can include harmless transformations such as:

text
Whitespace variation
Case variation
Encoding variation
Formatting changes
Character substitutions

The objective is to determine whether safety controls depend too heavily on surface wording.

66. Boundary Testing

Test around policy boundaries.

For example:

Architecture & Data Flow
Allowed request
 |
 v
Borderline request
 |
 v
Disallowed request

Check whether behavior changes predictably.

67. Regression Testing

Every discovered vulnerability should become a permanent test.

Architecture & Data Flow
Incident
 |
 v
Attack case
 |
 v
Regression test
 |
 v
Future releases

This prevents the same vulnerability from returning.

68. Security Gates

A release may require:

Mathematical Formulation
Critical vulnerabilities = 0
High vulnerabilities = 0
Cross-tenant leakage = 0
Unauthorized tool execution = 0
Safety threshold >= target

Fail the release if requirements are not met.

69. Defense-in-Depth

No single defense is sufficient.

Architecture & Data Flow
User
 |
 v
Authentication
 |
 v
Authorization
 |
 v
Input controls
 |
 v
Model
 |
 v
Output validation
 |
 v
Tool policy
 |
 v
Business validation
 |
 v
Audit

If one layer fails, another can limit the impact.

70. Security Gateway

A centralized gateway can enforce:

text
Identity
Policy
Rate limits
Data classification
Model routing
Tool permissions
Audit

This creates consistent controls across AI applications.

71. Policy Engine

A policy engine can answer:

text
Can this user access this data?
Can this model process this data?
Can this agent use this tool?
Can this action be executed automatically?

Conceptually:


🐍 PythonInteractive WebAssembly
decision = policy_engine.check(
 user=user,
 action=action,
 resource=resource,
)

The model should not make these authorization decisions alone.

72. Data Egress Controls

Control where information can go.

Architecture & Data Flow
Sensitive data
 |
 v
Egress policy
 |
 +--> Approved internal destination
 |
 X--> External destination

Useful for:

email
webhooks
external APIs
file downloads

73. Output DLP

Before sensitive outputs leave the system:

Architecture & Data Flow
Generated output
 |
 v
DLP scanner
 |
 +--> Safe --> Send
 |
 +--> Sensitive --> Block / redact / review

This is especially important for customer-facing and agentic systems.

74. Security Telemetry

Track:

text
Prompt injection attempts
Tool denials
Authorization failures
Sensitive-data detections
Abnormal token usage
Rate-limit violations
Model refusals
Security policy violations

Security telemetry should be correlated with request IDs.

75. Security Monitoring

A useful pipeline:

Architecture & Data Flow
Events
 |
 v
Detection rules
 |
 v
Anomaly detection
 |
 v
Alert
 |
 v
Investigation

Examples:

Mathematical Formulation
Repeated injection attempts
+
High token usage
+
Multiple tool denials
=
Potential abuse

76. Incident Response

For an AI security incident:

Architecture & Data Flow
Detect
 |
 v
Contain
 |
 v
Preserve evidence
 |
 v
Investigate
 |
 v
Remediate
 |
 v
Validate
 |
 v
Document
 |
 v
Regression test

77. Kill Switches

Critical systems should be able to disable risky capabilities quickly.

Architecture & Data Flow
Agent
 |
 v
Policy
 |
 +--> write_tools = enabled
 |
 +--> write_tools = disabled

A kill switch should be simple and independently operable.

78. Security Review Checklist

Before production:

text
[ ] Threat model
[ ] Trust boundaries
[ ] Identity
[ ] Authorization
[ ] Tenant isolation
[ ] Data classification
[ ] Prompt injection testing
[ ] RAG poisoning testing
[ ] Tool security
[ ] Agent limits
[ ] Output validation
[ ] DLP
[ ] Rate limits
[ ] Abuse monitoring
[ ] Supply-chain review
[ ] Incident response
[ ] Regression suite
[ ] Kill switch

79. Practical Project 1: Prompt Injection Test Harness

Build a test suite that sends controlled adversarial cases to an AI application.

Measure:

attack success rate
refusal correctness
data leakage
tool execution
false positives

Convert every important failure into a regression test.

80. Practical Project 2: RAG Security Scanner

Build a pipeline that:

Architecture & Data Flow
Documents
 |
 v
Security checks
 |
 +--> suspicious instruction
 +--> permission anomaly
 +--> poisoned content
 +--> sensitive data
 |
 v
Quarantine / approve

Keep security decisions separate from model-generated classifications where possible.

81. Practical Project 3: Agent Tool Security Gateway

Implement:

Architecture & Data Flow
Agent
 |
 v
Tool gateway
 |
 +--> Identity
 +--> Allowlist
 +--> Schema validation
 +--> Authorization
 +--> Budget
 +--> Audit
 |
 v
Tool

Test unauthorized and malformed tool calls.

82. Practical Project 4: SQL Agent Security

Build an analytics agent that:

generates SQL
parses SQL
blocks write operations
restricts tables
applies row/column permissions
logs queries
limits result size

Test malicious and accidental destructive SQL.

83. Practical Project 5: Multimodal Red-Team Suite

Test:

text
Image injection
OCR injection
Audio injection
Video injection
Malicious documents

Measure whether untrusted multimodal content can cause:

policy bypass
data leakage
unauthorized tool calls

84. Practical Project 6: AI Security Regression Platform

Build:

Architecture & Data Flow
Attack library
 |
 v
Automated runner
 |
 v
AI application
 |
 v
Security evaluator
 |
 v
Results database
 |
 v
Release gate

Track vulnerabilities across model and application versions.

85. Advanced Exercise 1: Enterprise Threat Model

Create a threat model for:

Architecture & Data Flow
Enterprise AI assistant
 |
 +--> RAG
 +--> CRM tools
 +--> Email
 +--> Web search

Identify at least:

10 assets
10 threats
10 controls
trust boundaries
high-risk attack paths

86. Advanced Exercise 2: Indirect Injection

Design a test where:

Architecture & Data Flow
Malicious webpage
 |
 v
Web retrieval
 |
 v
Agent
 |
 v
Tool

attempts to cause an unauthorized action.

Define:

expected behavior
detection
mitigation
regression test

87. Advanced Exercise 3: Cross-Tenant Attack

Design a test for:

Architecture & Data Flow
Tenant A user
 |
 v
Shared RAG system
 |
 v
Attempt to retrieve Tenant B data

Test:

direct retrieval
semantic search
cache
agent memory
citations
tool access

88. Advanced Exercise 4: Agent Privilege Escalation

Design an agent architecture with:

text
Read tools
Write tools
Admin tools

Then define which actions require:

no approval
user approval
administrator approval

Explain how privilege escalation is prevented.

89. Advanced Exercise 5: AI Supply-Chain Security

Create a security process for approving:

text
Model
Dataset
Container
Python package
Embedding model
Tool plugin

Include:

provenance
versioning
hashes
vulnerability scanning
license review
security approval

90. Advanced Exercise 6: Red-Team Program

Design a quarterly enterprise AI red-team program.

Include:

text
Threat taxonomy
Attack library
Automated testing
Manual testing
Production telemetry
Security regression
Executive reporting
Incident response

Define measurable security KPIs.

91. Common Mistakes

Mistake 1: Treating the system prompt as a security boundary#

Attackers can manipulate model behavior.

Mistake 2: Trusting retrieved content#

RAG content can contain malicious instructions.

Mistake 3: Giving agents excessive permissions#

Agent capability should be limited by policy.

Mistake 4: Executing model output directly#

Validate generated SQL, code, commands, and tool arguments.

Mistake 5: Ignoring caches#

Caches can become cross-user data-leak paths.

Mistake 6: Testing only single-turn prompts#

Real attacks can be multi-turn and contextual.

Mistake 7: Ignoring multimodal inputs#

Images, audio, and video can carry malicious instructions.

Mistake 8: No security regression suite#

A fixed vulnerability can return after a model or prompt update.

Mistake 9: No rate limits#

AI can be abused for denial of service and economic exhaustion.

Mistake 10: Trusting tool outputs#

External tools can return malicious or manipulated content.

Mistake 11: No supply-chain controls#

Models, datasets, packages, and containers are all dependencies.

Mistake 12: Red teaming only the model#

The application, data, tools, identity, and infrastructure must also be tested.

92. Final Mental Model

AI red teaming can be summarized as:

Architecture & Data Flow
 AI SYSTEM
 |
 v
 THREAT MODEL
 |
 v
 ATTACK SURFACE
 |
 +--------------+--------------+
 | | |
 v v v
 MODEL DATA TOOLS
 | | |
 Prompt RAG Agent actions
 attacks attacks tool abuse
 | | |
 +--------------+--------------+
 |
 v
 ADVERSARIAL TESTS
 |
 v
 EVALUATION
 |
 v
 MITIGATIONS
 |
 v
 SECURITY REGRESSION
 |
 v
 PRODUCTION
 |
 v
 MONITOR + LEARN

The deepest principle is:

AI security is a system property. A secure model inside an insecure application is still an insecure AI system.

93. Key Takeaways

AI red teaming tests how the complete AI system behaves under adversarial conditions.
The model is only one part of the attack surface.
System prompts are not security boundaries.
Direct and indirect prompt injection must both be tested.
Retrieved documents, websites, emails, tool results, and multimodal inputs should be treated as untrusted data.
RAG requires authorization-aware retrieval.
Tenant isolation must be enforced outside the model.
Agents should follow least privilege.
Tool calls require allowlists, schema validation, authorization, and business-rule checks.
High-impact actions should support human approval.
SQL, shell, browser, filesystem, and cloud tools require stronger controls.
Generated output is untrusted input and should be validated.
Caches and memory can become security boundaries.
Multimodal systems introduce additional injection paths.
AI systems can be abused for denial of service and economic exhaustion.
Models, datasets, packages, containers, and tools form an AI supply chain.
Every discovered security vulnerability should become a regression test.
Automated red-team pipelines make security validation repeatable.
Defense-in-depth is more reliable than any single AI safety mechanism.
Security telemetry should cover both attacks and attempted attacks.
Kill switches and incident runbooks reduce the impact of production failures.
The goal of red teaming is not merely to produce attacks; it is to discover weaknesses and turn them into durable engineering improvements.

94. Knowledge Check

Question 1#

What is AI red teaming?

Answer: Structured adversarial testing designed to discover realistic security, safety, privacy, and reliability weaknesses in an AI system before they are exploited.

Question 2#

Why is prompt injection dangerous?

Answer: It can manipulate model behavior and potentially influence retrieval, tool usage, or agent actions.

Question 3#

What is indirect prompt injection?

Answer: Malicious instructions embedded in external content such as a document or webpage that the AI later retrieves or processes.

Question 4#

Why can an agent be more dangerous than a chatbot?

Answer: An agent can translate model decisions into actions through tools and external systems.

Question 5#

What is least privilege?

Answer: Giving each component only the permissions required for its intended task.

Question 6#

Why should tool outputs be treated as untrusted?

Answer: Tools may consume external or attacker-controlled data, which can contain misleading or malicious instructions.

Question 7#

Why is tenant isolation important?

Answer: It prevents users or organizations from accessing another tenant's data through retrieval, caches, memory, tools, or application bugs.

Question 8#

What should happen after discovering a serious AI security vulnerability?

Answer: Contain and remediate it, validate the fix, and add a regression test so future releases are checked automatically.

Question 9#

Why is multimodal red teaming necessary?

Answer: Images, audio, video, and extracted OCR/transcription content can introduce attack paths that do not exist in text-only systems.

Question 10#

What is the central AI security principle?

Answer: Security must be enforced across the complete system through identity, authorization, validation, isolation, policy, monitoring, and defense-in-depth rather than relying on the model alone.

95. Course Progression

The course has now moved from reliability engineering into adversarial security validation.

Architecture & Data Flow
Enterprise Generative AI
 |
 v
AI FinOps & Cost Engineering
 |
 v
AI Reliability & SRE
 |
 v
AI Red Teaming & Security Testing
 |
 v
Future / Research AI Architectures
 |
 v
Full Generative AI Capstone

The next notebook moves into Future / Research AI Architectures, covering emerging model architectures, multimodal foundation models, mixture-of-experts evolution, state-space and recurrent alternatives, long-context systems, memory architectures, retrieval-native models, reasoning systems, world models, neuro-symbolic approaches, continual learning, test-time compute, agent-native architectures, embodied AI, efficient inference, future training paradigms, research evaluation, and how to reason about emerging AI architectures without chasing hype.

Knowledge Checkpoint

AI Red Teaming & Security Testing Checkpoint

Q1.What is an Adversarial Jailbreak in LLM red teaming?

ACrafted inputs (e.g. roleplay framing, base64 encoding, adversarial suffixes) designed to bypass alignment guardrails and force the model to generate restricted content.

BRooting a physical smartphone running AI software.

CCracking the password of a database.

DSpeeding up model inference.

Q2.What is Automated Red Teaming (e.g. PyRIT / GCG - Greedy Coordinate Gradient)?

AUsing automated optimization algorithms or attacker LLMs to iteratively probe target models for safety vulnerabilities at scale.

BRunning unit tests in Jenkins.

CScanning source code with ESLint.

DValidating CSS stylesheets.

Q3.What is a 'Crescendo Attack' in conversational AI red teaming?

AA multi-turn attack that starts with benign, innocent questions and gradually guides the model toward generating restricted material across several conversational turns.

BSending 10,000 requests per second to crash the API.

CInjecting SQL code into prompt headers.

DOverheating the GPU with heavy matrix math.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

Generative AI Security, Privacy, Governance & Responsible AI

Next Lesson

LLMOps, Inference Optimization & Production GenAI Systems