AI Security

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. NIST FIPS 199: Security objectives and losses

   Confidentiality preserves the restrictions that authorized parties place on information access and disclosure, including privacy and proprietary information. Integrity protects information against improper changes or destruction and includes authenticity and non-repudiation. Availability requires access and use to remain dependable and timely. Their corresponding losses are unauthorized disclosure, unauthorized modification or destruction, and disrupted access or use. Applied to a support application, exposing a private ticket breaks confidentiality, an unauthorized ticket edit breaks integrity, and making support records unavailable when needed breaks availability. One incident can damage several properties.

2. Stealing Machine Learning Models via Prediction APIs

   Model extraction aims to reproduce the target model function from query access, rather than recover a particular training record. Tramer and colleagues show that information-rich prediction APIs can expose enough structure to reconstruct or closely approximate studied models. Confidence outputs aid equation-solving for some model classes and reveal useful structure for others. Removing confidence values makes these attacks less efficient but does not eliminate label-only extraction in their experiments. The defended asset can therefore be the model behavior itself, even when the underlying service is not compromised.

3. NIST SP 800-30 Rev. 1: Guide for Conducting Risk Assessments

   NIST models adversarial risk by identifying threat sources and events, vulnerabilities or predisposing conditions, safeguards, likelihood, and impact. Threat-source characterization includes capability, intent, and targeting. Overall likelihood combines whether an adversary initiates an event with whether the initiated event produces adverse impact; risk then combines likelihood and impact. A useful AI threat model should therefore state the attacker’s required capabilities, the attack path and conditions, the affected asset, existing controls, and resulting harm rather than label model misbehavior as risk by itself.

4. NIST AI 100-2e2025: Adversarial Machine Learning

   An AI threat model identifies the protected property, attack stage, attacker knowledge, and what the attacker can control. Training-data control, model control, query access, and control of retrieved resources enable different attacks. Indirect prompt injection specifically exploits resource control: a third party changes material a legitimate user's application reads, allowing the data channel to influence instructions. Its consequences include corrupted outputs, lost availability, and private-data disclosure. This differs from poisoning training data and from a malicious user directly querying a model; the controls must match the access path.

5. Balancing Innovation with Security & Safety

   Test the same sensitive-data request under different user roles and target identities.

6. OWASP LLM10:2025 Unbounded Consumption

   Uncontrolled inference can consume shared capacity or money without crossing a content-policy boundary. Long inputs, repeated requests, and expensive operations can make request count a poor proxy for work. OWASP recommends input limits, per-user quotas, resource management, timeouts, throttling, graceful degradation, and bounds on queued and total actions. For an agent, the engineering implication is to account for the whole task, including generated tokens and downstream calls, and enforce limits before repeated work expands beyond its budget.

7. OWASP Threat Modeling Cheat Sheet

   Threat modeling examines a specific system from an adversarial perspective and develops responses to identified threats. A data-flow diagram represents processes, stores, external entities, information flows, and trust boundaries. The model should include cloud identities, managed services, and provider-versus-customer responsibilities. OWASP connects identified threats to actionable mitigation requirements and asks whether their success or failure can be tested. Removing an unnecessary feature is a distinct response from adding mitigations. The model should evolve with the system.

8. AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

   AgentPoison studies agents that retrieve stored demonstrations to guide future actions. Its threat model grants an attacker the ability to insert a small number of records into a memory or knowledge base, plus access to an embedder during attack construction. A trigger in a later query makes malicious demonstrations likely to be retrieved; those examples influence the model without changing its weights. The result is a persistent data-to-behavior attack surface: later retrieval can reactivate previously stored hostile material even when the current task is legitimate.

9. Invariant Labs: MCP Security Notification—Tool Poisoning Attacks

   Invariant's original experiments place malicious instructions in a tool description supplied by an attacker-controlled MCP server. In the reported Cursor demonstration, a nominal arithmetic tool induces sensitive-file reads and transmission through an additional argument. The report describes a confirmation interface that did not expose the complete sensitive input. A second experiment uses one server's description to redirect email sent through another server's legitimate tool; invoking the malicious tool is unnecessary. The authors also identify replacement of previously approved descriptions as a later compromise path.

10. Saltzer and Schroeder: Basic Principles of Information Protection

    Least privilege limits each program and user to the authority needed for its task, reducing the damage from error or compromise. Complete mediation requires authorization checks for every access, including lifecycle paths such as recovery, and reliable identification of the requester. Cached authorization decisions must account for changed permissions. Fail-safe defaults make access depend on explicit permission. Applied to an agent, these principles require enforcement where a proposed operation actually reaches a protected resource; a model promise or a tool description is not that enforcement.

11. NIST CSRC Glossary: Reference Monitor

    A reference monitor is a set of requirements for a mechanism enforcing access-control policy over subjects and objects. The mechanism must always be invoked, resist tampering, and be small enough for meaningful analysis and testing. Applied to AI tools, the protected operation must pass through enforcement outside the model on every access; an instruction, classifier, or schema-valid tool request cannot serve as the reference monitor if the model can bypass or modify it.

12. Explaining and Harnessing Adversarial Examples

    A model can classify ordinary test examples accurately yet fail on deliberately selected nearby inputs. Goodfellow, Shlens, and Szegedy show how many small input changes can accumulate into a large output change in a high-dimensional linear calculation; nonlinear networks can exhibit similar behavior. Adversarial examples can transfer across models, so an attacker need not always obtain the target weights. Their adversarial training mixes ordinary and adversarial examples and improves measured robustness without eliminating errors. Ordinary accuracy, confidence, and tested resistance to a particular perturbation method are distinct claims.

13. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

    BadNets studies an attacker-controlled training process that returns a model with good ordinary validation accuracy but attacker-chosen behavior when a trigger appears. Poisoned examples can teach the association between the trigger and an unwanted label. The paper also demonstrates a backdoor surviving transfer learning in its studied setting. This differs from evasion against an honestly trained model: the attacker has altered what the model learned. A behaviorally malicious weight artifact need not exploit a file loader, so clean validation accuracy and safe deserialization address different risks.

14. Simon Willison: Prompt injection attacks against GPT-3

    On September 12, 2022, Willison proposed the name prompt injection for attacks against applications combining instructions with untrusted text. His translation example illustrates the mechanism: text supplied for translation instead redirects the requested output. He also reports his own prompt-disclosure experiment. The practical concern was application developers treating natural-language input as passive data even though it could change model behavior.

15. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    Indirect prompt injection places attacker instructions in material an application is likely to retrieve, rather than in a direct user request. The paper demonstrates how applications can confuse external data with instructions, allowing retrieved text to redirect behavior or influence subsequent API calls. This makes the provenance and trust level of a context item separate from its relevance to a query: useful retrieved material can still be adversarial.

16. The Protection of Information in Computer Systems

    Saltzer and Schroeder describe complete mediation, fail-safe defaults and least privilege: check authority for accesses, deny absent permission, and limit a component’s granted powers. These principles apply during recovery as well as ordinary execution. In an agent loop, tool proposals and retrieved instructions must therefore pass an independently enforced authorization boundary before they can affect protected resources. Model capability does not establish the caller’s authority.

17. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    Indirect prompt injection places attacker-authored instructions in material an application retrieves at inference time, such as web pages or messages. When inserted into model context, that material can be interpreted as instructions and redirect subsequent answers or API use without the attacker controlling the user's request or model weights. The paper demonstrates attacks on integrated systems and develops a taxonomy including information theft and manipulated content. Engineering implication: retrieval and tool results remain untrusted data; the application must independently enforce resource access, tool permissions, transaction approval, and data-release restrictions. A model's interpretation of retrieved text cannot grant these privileges.

18. How to Build Trustworthy AI

    Prompt-injection exposure includes retrieved documents and scraped websites, as well as direct user messages.

19. Why, and how you need to sandbox AI-Generated Code? — Harshil Agrawal, Cloudflare

    Generated code can exhaust resources or expose credentials through ordinary mistakes and apparently helpful behavior; adversarial input adds another path to the same privileged execution.

20. OWASP SQL Injection Prevention: prepared statements and bound values

    Parameterized queries define SQL structure separately from the values bound into its placeholders. A supplied string is compared as a value rather than becoming new SQL syntax, even when it contains SQL-looking text. This relies on actual parameter binding, not concatenating input into the command. Parameters generally cannot stand for table names, column names, or sort keywords; those require application-controlled selection or redesign. The engineering comparison with prompts is precise: natural-language delimiters or Base64 marking alone do not supply an independently enforced parser contract that prevents marked content from influencing model behavior. Such marking can help a model recognize untrusted material without granting it SQL-style separation.

21. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

    The paper trains models to follow lower-priority instructions when compatible with higher-priority instructions and ignore conflicting ones. Context synthesis distributes compatible parts of a request across hierarchy levels; context ignorance trains against the answer expected without conflicting lower-priority material. GPT-3.5 Turbo experiments compare supervised fine-tuning and reinforcement learning from human feedback with and without hierarchy examples. Evaluations report improved resistance, including attack classes withheld from training, while testing benign instruction following separately. Two over-refusal tasks regress. The authors describe this as a model-based mitigation, acknowledge likely vulnerability to stronger attacks, and treat system-level safeguards as complementary. Learned role priority therefore is not an independent authorization check at a protected resource.

22. Building security around ML

    Encoding untrusted content does not by itself establish a reliable instruction boundary.

23. InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models

    The October 30, 2024 preprint investigates detectors that incorrectly classify harmless text containing attack-associated words as malicious. Its NotInject dataset contains 339 generated, manually reviewed benign examples containing such words. Evaluation separates ordinary benign inputs, malicious inputs, and these challenging benign cases. InjecGuard adds benign combinations of problematic tokens during training and reports improved balance across the categories. The study supplies a concrete reason to test legitimate security discussions and programming questions alongside attacks.

24. Simon Willison: The lethal trifecta for AI agents

    Simon Willison names three capabilities that jointly expose an agent to data theft: private-data access, exposure to attacker-controlled content, and an external communication channel. Untrusted material can influence a model that can fetch private information and then transmit it. External communication includes HTTP requests and image loads, not only an explicitly named email tool. The useful design question is where to break that path through capability separation or enforced outbound restrictions. Assess the combined workflow: separate tools can supply the three capabilities even when each tool has a legitimate purpose.

25. GitHub Copilot Chat: From Prompt Injection to Data Exfiltration

    The researcher reports placing instructions in source code analyzed by Copilot Chat. Those instructions caused generated Markdown to contain an image URL carrying information from conversation history. Automatic image retrieval then sent that information in an HTTP request. The disclosure therefore crossed two boundaries: source text influenced generation, and the renderer converted generated text into external communication. The report records submission to GitHub on February 25, 2024, fix confirmation on June 12, and publication on June 14.

26. From Arc to Dia: Lessons learned in building AI Browser

    The 'lethal trifecta' combines private-data access, exposure to untrusted content, and external communication in one assistant.

27. OWASP Server-Side Request Forgery Prevention Cheat Sheet

    Server-side request forgery can make an application act as a proxy to a service the attacker cannot reach directly. OWASP distinguishes systems that contact a known set of services from systems that must fetch arbitrary external destinations. It recommends application and network controls together. For known destinations, valid hostname syntax and membership in the permitted destination set are separate checks. Redirect following can bypass initial validation, and permitted domains can resolve to unexpected internal addresses. The guidance therefore addresses redirects and DNS behavior in addition to URL input.

28. AI Engineering with the Google Gemini 2.5 Model Family

    Function calling produces a requested function name and arguments; application code must dispatch the call and return its result to the model.

29. OWASP Input Validation Cheat Sheet

    Syntactic validation checks the form of structured fields; semantic validation checks whether values make sense in their business context, such as a start date preceding an end date. JSON Schema is one supported validation mechanism. Validation applies to potentially untrusted backend feeds and supplier data as well as direct user input. OWASP explicitly cautions that input validation is not the primary defense against SQL injection or cross-site scripting.

30. OWASP Access Control

    Authentication establishes identity; authorization decides which actions that identity may perform on particular resources. A user allowed to initiate a transfer must still be authorized for the source account. Least privilege limits the authority of running code and service accounts, while centralized checks reduce inconsistent enforcement. In an AI application, tool availability and a model-produced argument are therefore insufficient grounds to execute a business operation; the application must apply resource- and action-level policy.

31. OWASP Transaction Authorization Cheat Sheet

    Transaction authorization should let the user acknowledge significant transaction data, such as destination and amount, and bind approval to that transaction rather than an unrestricted session. The server controls authorization data and permitted state transitions. Changes to transaction data invalidate prior authorization or restart the process. Credentials should be unique per operation and valid only for a limited interval. A final server-side gate immediately tied to execution verifies that the transaction was properly authorized, preventing skipped checks and substitution between approval and use. For an agent, approving a draft action therefore must not silently authorize a changed target, payload, or scope.

32. OWASP: Transaction Authorization

    Approval must concern the significant transaction details the user actually reviewed. Store and verify those details server-side, protect them against substitution and enforce the authorization state sequence. If transaction data changes, invalidate the previous challenge or restart authorization. Credentials should be transaction-specific and time-limited. A final gate tied to execution verifies that this transaction was authorized. Application inference: bind approval to an immutable transaction version containing the action, target and material arguments, and reject execution if that version no longer matches.

33. MITRE CWE-367: Time-of-check Time-of-use Race Condition

    A time-of-check/time-of-use failure occurs when resource state changes after validation but before use, invalidating the earlier check. MITRE illustrates a privileged program checking permission for a filename and subsequently opening it after an attacker substitutes a different target. The permission check can behave correctly while the combined operation remains vulnerable. Mitigations include atomic operations or effective locking acquired before validation. Merely shortening the interval makes exploitation harder without fixing the underlying race.

34. Norm Hardy: The Confused Deputy

    Hardy describes a compiler that possessed both its caller's authority and additional permission to write system files. A caller supplied the billing file as the destination for debugging output, and the compiler overwrote it using its own stronger authority. The confused deputy is this failure to distinguish whose authority should apply to an operation. Hardy's capability proposal couples identification of a resource with permission to use it: the compiler receives a capability for its statistics file and separately uses the caller's capability for debugging output.

35. Securing Agents with Open Standards

    Scoped access still needs an explicit connection to the user on whose behalf the agent acts.

36. OWASP LLM06:2025 Excessive Agency

    Damaging agent actions can result from excessive functionality, permissions, or autonomy, whether the triggering model output is maliciously induced or merely mistaken. A summarizer does not need a send-mail operation; its downstream identity can also be read-only, and a separately permitted send operation can require approval. OWASP's complete-mediation recommendation places authorization in downstream enforcement, validating every request instead of asking the model whether it is allowed. Logging and rate limits can limit or reveal harm but do not remove excessive agency by themselves.

37. Unlock Agent Autonomy: The Runtime for AI-Native Systems

    Read-only access to an entire service may expose unrelated sensitive content; task-specific capabilities should restrict what can be read.

38. Your Insecure MCP Server Won't Survive Production — Tun Shwe, Lenses

    Expose narrowly defined outcomes rather than a broad catalog of underlying operations, and enforce permissions for individual tools and resources.

39. OWASP Transaction Authorization Cheat Sheet

    Users should identify and acknowledge significant transaction details before authorizing execution. The authorization sequence must not be skipped or reordered, and execution needs a final authorization check. Changing transaction data during authorization should invalidate previous authorization or restart the process. Applied to AI proposals, approval should authorize the inspected scope and values rather than an independently changing proposal.

40. Stripe API Reference: Idempotent requests

    Stripe uses an idempotency key to recognize retries and avoid repeating the same operation. Once endpoint execution begins, it stores the first response status and body, including failures, and returns that result for subsequent requests with the same key. Reusing a key with different parameters produces an error. Validation failures and concurrent conflicts before execution do not create a saved result. Keys may be removed after at least 24 hours; reuse after removal initiates a new request.

41. Beyond permission prompts: making Claude Code more secure and autonomous

    The engineering report separates filesystem isolation from network isolation. Restricting accessible paths limits local damage, while restricting outbound connections limits where processes can communicate. Either boundary alone leaves important attack paths: code with unrestricted network access can transmit readable secrets, while unrestricted filesystem changes can undermine containment. A sandbox defines where actions may occur; approvals remain a separate decision about whether a particular action is authorized. This is especially relevant when a coding agent reads repository content that may contain hostile instructions.

42. gVisor Security Model

    gVisor reduces exposure to the host kernel by intercepting application system calls in its Sentry and restricting the Sentry's own host-system interface. This protects a different boundary from an application's authorization policy. The documentation explicitly notes that an attacker may reach a vulnerable network service or another API without escaping a container at all. A sandbox therefore needs an architecture that constrains accessible files, credentials, services, and network paths; isolation of execution alone cannot decide which allowed business action should happen.

43. From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

    Namespaces and cgroups address resource isolation and noisy neighbors, but containers still expose the shared host kernel; seccomp narrows that exposure at a compatibility cost.

44. gVisor security model: containment and host dependencies

    gVisor reduces untrusted code’s ability to exploit the host system-call interface. Its Sentry implements application system calls instead of passing them directly to Linux, and itself uses a restricted host interface; the Gofer mediates filesystem access. This adds defenses against escape, not a guarantee of immunity. Sandboxed code can still access mapped files and allowed network connections. Hardware side-channel defenses depend on the host OS and platform; resource limits depend on host cgroups, and network policy needs separate enforcement. Host-networking and directfs settings change which host operations are available. Architectural implication: a local assistant’s sandbox protects the host from untrusted execution but does not establish confidentiality against a compromised host kernel or host components controlling its isolation and resources.

45. Realtime multiplayer, automation, and you!

    Agentic Workflows keeps secrets outside the agent's execution boundary and mediates credentialed service calls through an external component.

46. Cloudflare Dynamic Workers: Egress control

    Cloudflare documents an enforcement point outside dynamically supplied Worker code: globalOutbound intercepts its fetch and connect calls. Setting it to null blocks those calls while explicitly supplied service bindings remain usable. Alternatively, a loader-side gateway can inspect requests and attach credentials held outside the untrusted Worker. Blocking ordinary fetching can break compatibility with HTTP client libraries. The credential example forwards requests and adds a token for a selected hostname; its code does not itself authorize every operation or reject every other destination.

47. Why, and how you need to sandbox AI-Generated Code? — Harshil Agrawal, Cloudflare

    Treat unbounded execution as both a cost risk and a denial-of-service risk, and enforce explicit execution limits.

48. NIST Privacy Framework 1.0: lifecycle and minimized audit evidence

    The framework inventories data elements, processing purposes, actions, owners and flows. Policies define permitted uses and retention periods; the data lifecycle aligns with system development and operations. Authorizations must be maintained and revocable, access limited by least privilege, and deletion and destruction performed under policy. Audit records themselves must incorporate data minimization. Engineering application: define the decision evidence needed for review, its purpose, authorized readers, retention trigger and disposal method before logging. Retain the necessary decision, model and policy versions and relevant evidence without indiscriminately copying personal data into logs, prompts or backups. Where review requires sensitive evidence, constrain fields, access and retention rather than treating auditability as permission to keep everything. Assess removal and disclosure across downstream copies and service providers.

49. RedisVL Extensions: Semantic Cache and Message History

    A semantic cache returns an existing response when a new prompt is sufficiently similar, bypassing a fresh model call. RedisVL documents filters that scope those lookups by user or conversation to avoid sharing one user's cached answer with another. Message history similarly uses session tags to separate conversations. These are distinct from similarity thresholds: a threshold controls which questions match, while a filter or session scope controls which stored records are eligible.

50. Through the AI Fog: The architectural decision the next 24 months of agentic security depends on.

    A task-oriented agent may copy sensitive data into an untrusted store for convenience, creating an unknown attack surface outside established enterprise controls.

51. Membership Inference Attacks against Machine Learning Models

    Membership inference tests whether an already supplied record participated in training; it need not recover that record's contents. Shokri and colleagues study classification APIs exposing probability vectors. Their attacker knows input and output formats and either the training architecture and algorithm or has access to the training service. Shadow models provide examples with known membership for learning differences between member and nonmember predictions. Experiments include commercial prediction services and hospital-discharge data, where participation itself can be sensitive. Precision and recall assess membership decisions.

52. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures

    Model inversion uses observable model behavior, often together with other known attributes, to infer sensitive input features or construct an input associated with a target output. Fredrikson, Jha, and Ristenpart show how exposed confidence information strengthens such attacks in decision-tree and face-recognition experiments. Inversion is not the same objective as testing whether a candidate belongs to training data, copying the model function, or recovering an exact stored training record. What the API reveals and what the attacker already knows determine what inference is possible.

53. Extracting Training Data from Large Language Models

    Carlini and colleagues demonstrate recovery of verbatim GPT-2 training examples through generated outputs. The study separates generating candidate text from estimating whether it appeared in training, then validates selected findings against the original dataset. Membership inference asks whether a candidate was used in training; extraction seeks to recover the content itself. High model likelihood alone produces false positives and is not proof of membership. The language-model background also distinguishes next-token probabilities from the sampling process that chooses a continuation. Training-data memorization is a different leakage path from an assistant exposing a private document supplied at runtime.

54. Microsoft: Design a secure multitenant RAG inferencing solution

    Microsoft's architecture requires tenant and user authorization filters before retrieved information reaches model context, including retrieval during an agent loop. A tenant partitions customers or users according to the application's business model; membership in that tenant does not authorize every record within it. The proposed data-access API preserves caller identity, routes to the appropriate store, applies filtering, and records access. Separate stores simplify isolation but increase cost and operational overhead; shared stores require explicit tenant discrimination and user-level restrictions.

55. PyTorch Security Policy: using models securely

    PyTorch treats models as programs and warns that running an untrusted model can be equivalent to running untrusted code. It recommends separating weights from Python code, checking provenance, and using an isolated environment. Serialization format changes the attack surface: restricted weight formats support fewer behaviors, while flexible loading has more ways to execute or process unsafe content. Even a safer format does not eliminate downstream input-validation risks, and TorchScript inspection utilities may execute code. Model acquisition and inspection therefore belong in the software supply-chain threat model.

56. How to Build Trustworthy AI

    Unsafe model serialization can turn model loading into arbitrary code execution; the narrated ModelScan example exposes AWS credentials when the model loads.

57. Building security around ML

    Some model formats and framework conveniences can execute code or write files; verify provenance, scan artifacts, and isolate uncertain models.

58. SLSA v1.2: Verifying Artifacts

    SLSA artifact verification checks a provenance signature, binds the provenance subject to the artifact digest, recognizes the builder identity, and compares build type, source, and external parameters with expectations. Verification outside the registry can detect registry or transit tampering. The documented guarantees remain conditional: Build L3 assumes trust in the build platform and does not cover its malicious insiders, while SLSA does not yet cover every unintended-package selection attack. Model weights, adapters, tokenizers, and deployment bundles can use these controls when represented as software artifacts.

59. Building security around ML

    Dataset URLs can outlive their original owners or content; verify downloaded data against available provenance and checksums.

60. OWASP LLM05:2025 Improper Output Handling

    Model output becomes another input at the component that consumes it. OWASP identifies dangerous transitions such as passing generated text into a shell, interpreting it in a browser, executing generated SQL, or using it to construct file paths. Defenses depend on that destination: context-appropriate output encoding, parameterized database operations, and validation before backend use. This is a downstream interpretation problem, separate from whether the answer is factually accurate.

61. MCP Security Best Practices: token passthrough and confused deputies

    MCP's security guidance forbids accepting and forwarding a token without verifying that it was issued for the MCP server. Token audience separation prevents a proxy from treating a credential meant for another resource as authority to use its own facilities. Passthrough can bypass request validation or rate limits and obscure who performed an action in downstream logs. The confused-deputy example separately requires per-client consent rather than allowing a remembered third-party authorization to authorize an unrecognized client silently.

62. What does Enterprise Ready MCP mean?

    Access controls need complementary data-loss prevention to address users sending inappropriate data to MCP servers.

63. Your LLM Stack Is a 2008 Database With Better Marketing

    The talk's four-pillar defense-in-depth model combines infrastructure security, access control, runtime security, and operational practice.

64. Safety and security for code-executing agents

    Model-based monitors are useful for identifying sensitive actions, but the speaker does not consider them a substitute for deterministic controls or human review.

65. OWASP: prompt injection and evaluator trust boundaries

    Indirect prompt injection occurs when externally supplied content is interpreted as instructions that redirect model behavior. Captured prompts, retrieved passages and tool results become the same attack surface when an evaluator reads them: an embedded instruction can attempt to alter a score, disclose context or invoke a connected action. OWASP recommends separating untrusted content, validating outputs in deterministic code, minimizing privileges, keeping credentials and functionality under application control, and requiring approval for high-risk operations. Applied to evaluators, isolate scoring from production credentials and write-capable tools. Authorize downstream actions independently of the evaluator's text or score.

66. NIST AI Risk Management Framework Core

    The AI RMF ties deployment decisions to documented context, risk tolerance, measured conditions, residual risk, responsible owners, and available resources or alternatives. It calls for documenting test conditions and limits on generalization, evaluating security and resilience, tracking emergent risks, and regularly monitoring third-party resources and deployed systems. Risk responses may mitigate, transfer, avoid, or accept risk; high-priority risks and unmitigated residual risks remain documented. Changes in methods, contexts, risks, or stakeholder needs require continued review, with mechanisms to deactivate systems whose outcomes conflict with intended use.

67. Microsoft Foundry: AI Red Teaming Agent concepts

    Microsoft’s current red-teaming documentation distinguishes model-output checks from agent tests involving tool outputs, prohibited actions, sensitive-data leakage, task adherence, and indirect injection through retrieved emails or documents. Its sensitive-data tests use synthetic records and mock tools; documented limitations include single-turn English scenarios, no training-set or memory leakage, no live production data, incomplete tool support, and no fully locked-down sandbox. Attack Success Rate is calculated over generated attacks, while model-based assessment can be nondeterministic and produce false positives requiring review.

68. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

    AgentDojo implements mutable environments containing dummy emails, calendars, documents, and other application data. Tools read and change that state. User tasks and attacker objectives have separate checks over outputs and environment state, allowing evaluation of completed actions rather than response tone alone. Its reporting separates benign utility, legitimate-task success under attack without adversarial side effects, and targeted attacker success. When evaluating a collection of attacks, a case counts as compromised if any attack succeeds. The authors warn that an LLM simulating or judging the environment could itself follow injected instructions.

69. Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd

    Require an executable proof of vulnerability and grade its observed effect deterministically.

70. Mitigating the risk of prompt injections in browser use

    Anthropic describes browser-agent defenses that combine robustness training, classifiers over untrusted context, and expert red teaming. Its reported evaluation uses an adaptive attacker with a budget of 100 attempts per environment, illustrating why one fixed malicious prompt is not a complete security test. The relevant measurement is whether an attacker achieves a defined objective against a specified application configuration and attack budget. The company explicitly warns that a low observed success rate still represents meaningful risk and does not establish immunity.

71. OWASP Logging Cheat Sheet

    Security records should identify event time, application and version, actor, action, affected object, result, and reason, with interaction identifiers connecting related events. Relevant events include authorization failures, sensitive-data access, exports, configuration changes, network failures, and attempts to bypass workflow order or limits. Evidence arriving from another trust zone may be missing, forged, modified, or replayed. OWASP recommends validating and sanitizing event data, restricting log access, excluding or protecting secrets, and testing logging failures and resource exhaustion. Security monitoring should be proportionate to identified risks rather than collecting an indiscriminate checklist.

72. It's 10pm. Do You Know Where Your Agents Are?

    Logging credential usage alone may leave operators unable to identify the actor or delegating user.

73. OpenTelemetry: Context propagation

    Context propagation connects work across service boundaries by carrying trace and parent-span identity to the receiving process. With W3C Trace Context, the sender injects traceparent and the receiver extracts it so downstream spans belong to the same request. Trace and span identifiers can also correlate logs with that execution. Without propagation, individually instrumented services can still produce disconnected records. Incoming trace context is not inherently trustworthy, and propagated baggage can leak sensitive information; correlation metadata needs an explicit trust boundary too.

74. NIST SP 800-61r3: incident response and verified recovery

    Triage validates an incident report and estimates severity and urgency; response priority considers impact, scope, and available resources. Containment limits expansion, while eradication removes persistence mechanisms and entry points. Investigators preserve the integrity and provenance of evidence and action records, with restricted access and defined retention. Recovery can begin during response under explicit criteria. Verify restoration assets before use, check restored systems for compromise, remediate root causes, and verify restoration before production use. Confirm service restoration with owners and monitor its adequacy. For an AI deployment-state example, specify affected components, containment, preserved evidence, authorized recovery actions, trusted restoration inputs, verification criteria, and who confirms resumed service; these are application choices implementing the guidance.

75. Balancing Innovation with Security & Safety

    Monitor responses under changing inputs and dependencies, confidential-data egress, and failure rates with tolerance-based alerts.

76. NCSC: Secure operation and maintenance of AI systems

    Security monitoring must observe both abrupt and gradual changes in model and application behavior, which may arise from compromise or ordinary drift. NCSC recommends monitoring inputs with privacy constraints so operators can investigate and remediate misuse. Its update guidance explicitly treats changes to data, models, or prompts as possible behavior changes that must be reflected in testing and evaluation. Security evidence therefore has a version and configuration: a tested deployment is not automatically evidence for the next model or prompt revision.

77. RFC 7009: OAuth 2.0 Token Revocation

    After validating the requesting client and token relationship, the authorization server invalidates the token. The RFC requires immediate invalidation but recognizes propagation delay among servers and requires implementations to minimize that window. Clients must stop using the token after HTTP 200; that status also covers an already invalid token. Revoking a refresh token should invalidate access tokens from the same grant when access-token revocation is supported. Revoking an access token may also revoke its refresh token; related-token and grant effects depend on server policy. Revocation removes future credential use. The protocol contains no application rollback operation, so it does not establish reversal of a sent message, disclosed data, or another completed action.

78. AI’s Jurassic Park Period

    Evaluate metadata changes against the actual evidentiary question, and record unavoidable transformations rather than treating every acquisition constraint identically.

79. Google Machine Learning Glossary: training, inference, tokens, context, and RAG

    Training adjusts learned parameters using examples; inference applies a trained model to new input. A classifier predicts a category, whereas a generative language model produces a response as tokens, which may represent words, word pieces, or characters. The context window limits how much tokenized information a model can process. Retrieval-augmented generation retrieves material after training and supplies it with the request; adding a retrieved document to context is not itself an update to model weights. These distinctions locate different security entry points in the training pipeline and the running application.

80. Dia Security Bulletins: Security Story of fetch_web_content

    Dia's February 12, 2026 retrospective describes a web-fetch tool that could transmit private context through model-generated URL parameters. The team reports detector bypasses and false alarms, and judged per-fetch confirmations too burdensome. It removed the feature before the June 2025 public beta and restored it about two months later with URL-provenance enforcement. The documented rule rejects invented URLs and query parameters while allowing URLs already present in supplied browsing context. This preserves fetching existing links while restricting arbitrary model-constructed requests.

81. Model Context Protocol: Cancellation, 2025-11-25 specification

    This MCP revision provides optional cancellation notifications identifying an in-progress request. Receivers should stop processing and release associated resources, but may ignore cancellation if processing has completed, the request is unknown, or it cannot be cancelled. Network latency can deliver cancellation after completion. Task-augmented requests use a separate cancellation mechanism. Consequently, sending a cancellation notification does not establish that execution stopped or resources were reclaimed.

82. Build Systems, Not Code

    Enforce idempotency through recorded system state rather than trusting the model to recognize a repeated task.

83. Build Systems, Not Code

    Treat external content as evidence rather than instructions, and gate consequential actions behind human approval.

84. Defeating Prompt Injections by Design

    CaMeL separates trusted-request planning from a quarantined model that parses untrusted content. Its interpreter stores tool results in variables hidden from the planner, tags values with sources and allowed readers, and tracks dependencies through transformations. STRICT mode also tracks conditional and loop dependencies. Before a tool runs, policies inspect its arguments and their dependency graph. This protects data flow as well as control flow. When an exception prompts regenerated code, existing variables, dependencies, and capabilities remain in interpreter state; untrusted-derived exception text is redacted. That documented retry continuity does not establish durable metadata preservation across independent executions. The paper provides no atomicity or rollback for operations that already caused side effects.

85. Unlock Agent Autonomy: The Runtime for AI-Native Systems

    Intent-based access should evaluate a requested capability against user intent, task intent, and context, with denial or human escalation for unjustified requests.

86. From Arc to Dia: Lessons learned in building AI Browser

    Dia asks users to review and confirm plain-text data before writing it into a form, treating confirmation as a product safety control rather than prompt-injection prevention.

87. It's 10pm. Do You Know Where Your Agents Are?

    RFC 8693 Token Exchange is used to combine delegated user authority with runtime identity in a per-tool authorization request.

88. We Gave an Agent Production Code Access and Then Tried to Sleep at Night

    Separate agent reasoning from privileged orchestration so an injected agent cannot directly push changes, open PRs, or trigger CI.

89. Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd

    Build a curriculum on two axes: target difficulty and exploitation difficulty.