Course Outline
Module 1 — Understanding AI Application Vulnerabilities
Lab: None — Architecture walkthrough & discussion
Developing a mental model of the attack surface for builders.
Topics:
- LLM, RAG, and agent architectures from the developer's perspective
- The request/response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Entry points for untrusted data into (and back into) the model
- Trust boundaries owned versus inherited by developers
- Why AI attacks rely on semantics rather than syntax
- Mapping OWASP LLM Top 10 to source code
Key insight: Any point where untrusted text reaches the model, or model output reaches your code, represents a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The "SQL injection moment" for AI—though complete avoidance is impossible.
Topics:
- Differences between direct and indirect prompt injection
- Hidden instructions embedded in documents, web pages, or tool outputs
- Jailbreaks and role-confusion techniques
- The importance of separating instructions from data
- Defensive prompt design (delimiters, structure, minimal authority)
- Why prevention is partial—designing for containment instead
Hands-on exercises:
- Attack your own chatbot
- Bypass naive filters
- Restructure prompts to minimize blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The vulnerability class developers often underestimate.
Topics:
- Viewing model output as untrusted input for the rest of the application
- Insecure output handling (LLM02): downstream XSS, SSRF, command/SQL injection risks
- Avoiding direct eval/exec/render of raw model output
- Utilizing structured outputs and schema validation
- Output encoding and allowlisting strategies
- Safe rendering practices in web/UI contexts
Hands-on exercises:
- Identify and remediate an insecure-output-handling vulnerability
- Enforce a JSON schema on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
One of the most significant new attack surfaces—and one you build.
Topics:
- Threats targeting vector databases and retrieval mechanisms
- Ingestion sanitization techniques
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions within retrieved content (indirect injection)
- Data exfiltration risks via retrieval processes
Hands-on: Poison a RAG pipeline with a malicious document; implement ingestion sanitization and retrieval scoping to defend against it.
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
The stage where vulnerabilities translate into actions.
Topics:
- Excessive agency (LLM06) and tool abuse
- Implementing least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop mechanisms
- Sandboxing tool execution environments
- Scoped, short-lived credentials for agents
- Limiting autonomous loops and chaining behaviors
Hands-on exercises:
- Restrict permissions on an overly permissive agent
- Add an allowlist and approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost Management
Lab: Lab 05 — 05-Secrets-and-Cost
Operational oversights that cause the most immediate damage.
Topics:
- API key and secret management (never embed in prompts, code, or logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval processes
- Mitigating denial-of-wallet attacks via unbounded token/cost consumption
- Implementing rate limits, token budgets, and timeouts
- Logging practices that prevent secret or PII leakage
Hands-on exercises:
- Remove secrets from prompt/code paths
- Add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Evaluating buy vs. build for input/output safety.
Topics:
- Capabilities and limitations of guardrail frameworks
- Input guardrails: injection, PII, and topic classifiers
- Output guardrails: validation, filtering, and grounding checks
- Distinguishing when to use guardrails versus deterministic checks
- Layering guardrails with controls from previous modules
- Managing performance, false positives, and failure modes
Hands-on exercises:
- Implement an input/output guardrail layer for an AI feature
- Analyze what the guardrail catches versus what it misses
Module 8 — Red-Teaming Your Own Application
Lab: Lab 07 — 07-Red-Teaming
Deploy with the assumption that an attacker has already compromised it.
Topics:
- Constructing abuse and test suites for AI features
- Automated prompt injection and jailbreak testing
- Regression testing of guardrails and policies
- Integrating AI security checks into CI pipelines
- Managing model and dependency supply chain (provenance, pinning)
- A pre-deployment security checklist for AI features
Hands-on exercises:
- Write automated red-team tests for an AI feature
- Integrate these tests into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: None — Scoring exercise (utilizes the Capstone app)
Transform your implementation into a repeatable score.
Topics:
- The AI Security Hexagon: six core questions instead of "is it secure?"
- Six scored categories: Data, Prompt, Agent, Supply Chain, Detection, Governance
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- Understanding the Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Using pre- and post-hardening scores as metrics
Hands-on exercises:
- Score the Capstone application on the 100-point scale
- Identify the single change that most significantly improves the score
Key insight: The three highest-weighted categories correspond to trust boundaries owned by developers—meaning the score accurately measures what this course has taught.
Capstone Project
Participants harden a deliberately vulnerable AI application from end to end.
The starter application includes:
- An injectable prompt
- Insecure output handling
- An unscoped RAG pipeline
- An over-permissioned agent
- Secrets embedded in the prompt path
- Absence of cost limits
Participants apply course concepts to:
- Restructure prompts for containment
- Validate and encode model outputs
- Sanitize and scope retrieval processes
- Apply least privilege and approval gates to the agent
- Relocate secrets and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened application plus a short OWASP LLM Top 10 self-assessment.
Module - Lab Mapping
Labs proceed in order, mirroring the module sequence. The course comprises 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion, and Module 9 is a scoring exercise; therefore, neither has a dedicated lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab—it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder—it runs as a scoring exercise against the Capstone app.
Requirements
- Skill Level: Intermediate.
- Prerequisites include proficiency in building and consuming REST APIs, experience with a scripting language (Python is used in labs), fundamental application authentication concepts, Git, and CLI usage.
- No machine learning background is necessary—this course focuses on application security for those who build with LLMs, not train them.
Target Audience
- Software and backend engineers developing LLM features
- Full-stack and API developers
- AI/ML application engineers
- Platform engineers deploying copilots and agents
- Tech leads and senior engineers responsible for AI features
Testimonials (2)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us
