Structured Outputs and Tool Calling

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. Function Calling is All You Need

   Raw function calling expresses the model's intended action; application code must execute it. In the speaker's API terminology, tools are a broader category that also includes hosted capabilities.

2. Your Agent Didn’t Fail. Your Harness Did.

   Internal acceptance does not prove the intended result appeared at the user-visible boundary.

3. JSON Schema: Object reference

   JSON syntax only establishes that a value can be parsed. JSON Schema adds constraints on the value’s structure. For objects, properties describes allowed field schemas but does not by itself require those fields; required names mandatory fields. Additional properties are allowed by default unless constrained. A missing field and a present null value are different. Composition matters: additionalProperties recognizes properties in its own subschema, whereas unevaluatedProperties can account for evaluated fields across subschemas in supporting drafts.

4. JSON Schema 2020-12 Validation Primitives

   The type keyword accepts a type name or an array of unique type names; an instance must match at least one named type. Names are null, boolean, object, array, number, string, and integer. integer accepts numbers with zero fractional part. Validation does not prescribe coercing a numeric string into a number. enum accepts only values equal to an element of its array, which can contain different JSON types. Numeric minimum and maximum are inclusive; exclusiveMinimum and exclusiveMaximum are strict: x >= minimum, x <= maximum, x > exclusiveMinimum, x < exclusiveMaximum. multipleOf requires a positive divisor and an integral quotient. Array minItems and maxItems are nonnegative integer bounds on length; uniqueItems=true requires all elements to be distinct.

5. JSON Schema Dialect Declaration

   A dialect defines which schema keywords and semantics apply. A root $schema declares that dialect using its meta-schema identifier; for example, https://json-schema.org/draft/2020-12/schema identifies Draft 2020-12. The meta-schema describes valid schemas, whereas the application schema describes valid instance data. Without $schema, an implementation may use external configuration or assume a dialect. Externally referenced schema documents declare their own dialect rather than inheriting the referring document's declaration.

6. JSON Schema: Annotations

   JSON Schema's default keyword is an annotation; validation does not insert its value into missing fields. Documentation and form generators may use it as a hint. Descriptions explain the purpose of data but do not themselves enforce constraints. A default should satisfy its associated schema, although the annotation rules do not require that.

7. JSON Schema Array Items and Tuples

   JSON arrays are ordered and may contain mixed types. For a homogeneous list, items applies one schema to every element; one failing element invalidates the array. An empty array satisfies this element constraint unless a length or other constraint excludes it. In Draft 2020-12, prefixItems assigns schemas by position, while items constrains the remaining elements. prefixItems alone neither requires every listed position nor forbids additional elements. For an exact-length tuple, combine positional schemas with a minimum length and exclusion of extra items. contains requires an element matching its subschema; minContains and maxContains constrain the number of matches.

8. Structured model outputs — OpenAI API

   The API guide distinguishes schema-adherent structured output from JSON mode, which guarantees JSON syntax but not adherence to a specific schema. It distinguishes response formatting from function calling and documents a supported subset of JSON Schema. Applications must handle refusals and incomplete responses rather than assuming every response parses into the requested object. The guide explicitly warns that structured outputs can contain mistakes. Typed SDK helpers can connect application types and schemas, but successful parsing remains only one stage in a reliable consumer.

9. Structured model outputs: supported contracts and incomplete responses

   The API distinguishes structured response formats from function calling. Structured Outputs supports a subset of JSON Schema, and unsupported schemas can be rejected. Applications must inspect completion status and handle refusals or output-token exhaustion rather than blindly treating every response as a completed record. The guide also states that schema-conforming outputs can contain mistakes. Streaming lets consumers process generated structure incrementally but does not make a partial prefix an authorized or semantically validated action.

10. Revised Report on the Algorithmic Language ALGOL 60

    The revised ALGOL 60 report defines program syntax with recursive metalinguistic formulas: named syntactic variables, ::= for definition, | for alternatives, literal marks, and juxtaposition for sequence composition. This demonstrates an early continuing role for formal grammars: specifying which symbol sequences constitute a language independently of executing the processes those sequences describe.

11. Implementing Remote Procedure Calls — Birrell and Nelson

    Birrell and Nelson’s February 1984 paper describes RPC as passing arguments across a network, executing a procedure remotely, and returning its results. Their Xerox Cedar implementation generated client and server stubs from Mesa interface modules listing procedure names and argument/result types. The stubs packed and unpacked values and dispatched incoming calls. This reduced the communication code application programmers needed to write, while leaving fundamental difficulties such as independent component failures.

12. JSON Schema Media Type — draft-zyp-json-schema-01

    Kris Zyp's December 5, 2009 Internet-Draft described JSON Schema as a JSON-based contract for required application data, validation, documentation, navigation, and interaction. It separated a schema from the instance it describes and included constraints for types, properties, arrays, numeric bounds, and enumerated values. This establishes a dated early standards proposal for machine-readable JSON contracts.

13. A Syntactic Neural Model for General-Purpose Code Generation

    Yin and Neubig’s 2017 model translated natural-language descriptions into abstract syntax trees: tree representations of a program’s grammatical structure. Generation selected grammar-production actions or emitted terminal tokens, then deterministically converted the tree into code. Encoding syntax in the available actions relieved the model of learning the entire grammar from limited training data. This provides an earlier example of enforcing structure during generation rather than checking only the finished text.

14. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

    PICARD addressed invalid SQL generated by language models through incremental parsing during decoding. At each step it checked high-probability candidate tokens and rejected inadmissible continuations. Unlike approaches requiring specialized decoder architectures or control vocabularies, PICARD operated on ordinary generated text and could be enabled at inference time. Its checks could also use the supplied database schema to reject nonexistent table or column names. The authors evaluated fine-tuned T5 models on Spider and CoSQL.

15. Toolformer: Language Models Can Teach Themselves to Use Tools

    Timo Schick and colleagues' February 2023 Toolformer paper trained GPT-J to decide which of five APIs to call, when to call them, what arguments to supply, and how to incorporate returned results. Candidate calls were sampled from a few demonstrations, retained when they improved future-token prediction under the paper's loss criterion, and used for fine-tuning. The experiments included calculator, question-answering, search, translation, and calendar tools.

16. Function calling and other API updates — OpenAI

    On June 13, 2023, OpenAI announced function calling for its GPT-4-0613 and GPT-3.5-Turbo-0613 API models. Developers supplied function descriptions through JSON Schema, and models selected functions and produced argument objects. The announcement presented both external-operation uses, such as weather lookup and email, and structured extraction through an extract_people_data signature. Thus function-shaped output served both application data extraction and proposed operations.

17. Efficient Guided Generation for Large Language Models

    Brandon Willard and Rémi Louf's July 2023 paper formulates regular-expression guidance as transitions in a finite-state machine and precomputes an index from recognizer states to acceptable model-vocabulary tokens. This moves repeated validity work outside token sampling and replaces a full-vocabulary validity scan at every step with an average constant-time lookup in the analyzed construction.

18. Transformers Generation Configuration and Termination

    With one beam, do_sample=false selects greedy decoding and do_sample=true selects multinomial sampling. Greedy decoding chooses a maximum-probability next token; sampling draws a token according to the selection distribution. Transformers documents termination through end-of-sequence token IDs, configured stop strings, and generation-length limits. max_new_tokens counts generated tokens without counting the prompt. These conditions control when generation stops, rather than checking that a structured record is complete. Application inference: a length cutoff or stop-string match can leave an unfinished string, array, or object; termination alone is insufficient evidence that parsing and validation will succeed.

19. Python json: Decoder behavior and interoperability

    Python’s JSONDecoder.raw_decode returns both a decoded value and the position where its JSON document ended, explicitly permitting callers to process strings with trailing material. The default decoder also accepts NaN and infinities outside JSON’s number grammar. Its strict option controls unescaped control characters inside strings; it is not a general switch for every interoperability restriction. parse_constant can reject nonstandard numeric constants, and object_pairs_hook exposes member pairs for custom handling.

20. RFC 8259: JSON object member uniqueness

    JSON object names should be unique. For duplicate names, implementations may retain the last value, reject the object, or expose every pair. Successful parsing therefore does not establish a consistent interpretation across consumers.

21. Pydantic: JSON parsing and partial JSON parsing

    Pydantic publishes an incomplete-array example whose partial parser returns the two completed strings while discarding an unfinished third string; ordinary parsing raises an end-of-input error. Another example validates a model from an incomplete object because the retained fields satisfy that model. Thus obtaining a parsed, validated object does not establish that the original response finished. The page also demonstrates that strict validation accepts some representations when parsing JSON directly that it rejects as Python objects, including date strings and arrays representing tuples.

22. TypeScript Handbook: The Basics

    TypeScript removes type annotations when producing JavaScript; the annotations do not change runtime behavior. Its documented compilation example retains the function body but removes parameter types. Application implication: annotating a parsed model response with a TypeScript interface does not install runtime checks on incoming values.

23. Introducing Structured Outputs: dynamic constrained decoding

    OpenAI describes compiling supported JSON Schema into a context-free grammar and using a cached artifact to determine valid continuations after each generated token. Invalid next-token choices are masked during sampling. This differs from generating arbitrary text and only then parsing or validating it: constraints narrow the generation path itself. The mechanism enforces represented structure, not the factual correctness of field values. The announcement explicitly identifies refusal and early stopping as cases requiring separate handling.

24. No more bad outputs with structured generation

    Outlines filters candidate tokens during generation rather than relying only on instructions to produce valid output.

25. JSON String Escapes and Structural Recognition

    JSON strings begin and end with quotation marks. Quotation marks, backslashes, and control characters U+0000 through U+001F must be escaped inside strings. Legal escape forms include backslash followed by a designated character, or backslash-u followed by four hexadecimal digits. Objects and arrays recursively contain values. Application inference: a constrained decoder must retain whether it is inside a string, after a backslash, or partway through a Unicode escape across token boundaries. An escaped quotation mark does not end the string, and a brace inside a string does not close an object. Structural recognition therefore cannot be implemented by counting raw brace characters alone.

26. XGrammar Token Alignment and Cached Masks

    XGrammar documents a byte-level pushdown automaton: a tokenizer token can span grammar boundaries or split a UTF-8 character, so its bytes must be matched through successive transitions. Rule entry pushes stack state; completion pops it. Finite-state recognition has bounded state, while the added stack supports recursive nesting. Preprocessing builds an optimized automaton and caches context-independent token validity keyed by the current stack-top node; context expansion precomputes possible following contexts. Cache storage uses accepted/rejected token lists or bitsets plus context-dependent token lists. Each generation step retrieves cached validity, checks context-dependent tokens against live stacks, unions accepted tokens across ambiguous stacks, and masks invalid logits to negative infinity. After selection, the token advances the live parser state. Persistent stacks share structure to reduce copying. This caches neither every possible full stack nor all future work.

27. Grammar-Aligned Decoding

    Ordinary grammar-constrained decoding keeps any token whose prefix has at least one valid completion, then renormalizes among those tokens. This is not generally the model’s distribution conditioned on producing a complete valid string. Exact conditioning also weights each next-token choice by the probability that its future continuation will satisfy the grammar. Grammar-Aligned Decoding formalizes this distinction; ASAp approximates future grammaticality using accumulated samples. The difference explains how enforcing syntax can change content probabilities even when every completed output is grammatical.

28. Gemini API: Structured outputs

    Google's current guide distinguishes structured final responses from function calling used to request application actions. It documents only a subset of JSON Schema, warns that very large or deeply nested schemas may be rejected, and tells applications to validate values and handle schema-compliant but semantically incorrect output. The page separately labels combining structured outputs with tools as preview functionality limited to specified Gemini 3 models.

29. llama.cpp: Grammars and JSON Schema conversion

    llama.cpp supports GBNF grammars and conversion of a subset of JSON Schema into grammar constraints. Its documentation warns that unsupported schema features may be skipped silently and recommends inspecting converter warnings and testing the resulting grammar. The schema used for output constraints is not automatically inserted into the model prompt, so the intended content and structure still need explanation to the model; tool-calling schemas have different prompt handling. A valid application schema is therefore not evidence that a particular runtime enforces every keyword.

30. Generating Structured Outputs from Language Models: Benchmark and Studies

    Saibo Geng and colleagues' February 2025 JSONSchemaBench study evaluates constrained decoders separately on efficiency, schema-feature coverage, compliance, and task quality. Its processed collection contains 9,558 schemas from function-call, API, configuration, and repository sources. The authors distinguish accepting a schema, producing compliant samples, and exactly implementing the schema's accepted language; finite generated samples cannot establish the last property.

31. The Hidden Cost of Structure: How Constrained Decoding Affects Language Model Performance

    Maximilian Schall and Gerard de Melo's September 2025 study tested 11 models and found that structural constraints and task quality can move differently. In their experiments, instruction-tuned models often lost quality on generation tasks under constrained decoding while classification behavior was more stable; base models showed a different pattern. The result supports evaluating semantic task performance separately from format compliance.

32. OWASP Input Validation Cheat Sheet

    OWASP distinguishes syntactic validation of field representation from semantic validation of values in their business context. Its examples include ensuring a start date precedes an end date and checking a price against an expected range. Validation applies to potentially untrusted backend feeds as well as user-facing inputs. Recommended mechanisms include schema validation, allowed-value checks, and bounds on values and lengths.

33. Pydantic is STILL all you need

    Model-level validators can enforce relationships between fields, not just individual field types.

34. Design by Contract Introduction

    A precondition states what the caller must establish before invoking a routine. Given that condition, the implementation owes the postcondition on successful completion; it may assume the caller fulfilled the precondition. A postcondition can relate resulting state to entry state, such as count=old(count)+1 and lookup(key)=insertedValue. A class invariant expresses consistency constraints across the class's operations, such as 0≤count≤capacity. An output schema can check declared structure and value restrictions, but ordinarily does not establish that the caller was entitled to invoke the operation, that persistent state changed correctly, or that cross-operation invariants hold.

35. NIST SP 800-162: Guide to Attribute Based Access Control Definition and Considerations

    NIST defines attribute-based access control as deciding whether requested operations are allowed by evaluating attributes of the subject, protected object, requested operation, and sometimes current environment conditions against policy, rules, or relationships. Applied to a model-produced action request, valid argument shape supplies only part of the decision; current actor, target resource, operation, conditions, and policy remain independent authorization inputs.

36. Human seeded Evals — Samuel Colvin, Pydantic

    State known semantic constraints in field descriptions instead of relying solely on corrective retries.

37. Human seeded Evals — Samuel Colvin, Pydantic

    Returning validation errors to the model can turn a failed extraction into a successful retry.

38. "I've never seen anything scarier than an LLM with tool calls." — Erik Meijer aka @HeadinTheBox

    The speaker argues that broad judgments such as whether an answer is safe or a question is proper lack the mathematical specification needed for formal proof.

39. Tools — Model Context Protocol specification 2025-06-18

    MCP tools expose names, descriptions, and input schemas, with optional output schemas for structured results. Tool execution errors can be returned with isError, distinct from protocol errors. The specification requires servers to validate inputs and implement access controls, and recommends client-side result validation and tool timeouts. Tool annotations are untrusted unless supplied by a trusted server. Standardizing discovery and invocation does not itself grant authority: the host still decides what may execute and when a human can deny a proposed action.

40. Tool Calling Is Not Just Plumbing for AI Agents

    Use simple names, instruction-bearing descriptions, and explicit input and output schemas to make tool use and composition understandable.

41. Agentic Search for Context Engineering

    Start with a tool's core purpose, then add use conditions, exclusions, and prerequisites when routing becomes unreliable.

42. Writing effective tools for agents — with agents

    Anthropic recommends tools with distinct purposes and explicit parameter meanings rather than mechanically wrapping every API endpoint. Its examples contrast searching contacts with returning an entire address book, and consolidate related customer information into one contextual lookup. Tool descriptions should explain specialized terminology, resource relationships, and expected inputs and outputs. The team reports correcting a web-search behavior that unnecessarily appended 2025 to queries by improving the description. Error responses should identify actionable corrections rather than expose only opaque codes.

43. AI Engineering with the Google Gemini 2.5 Model Family

    Making many tools easy to attach can create a tool-selection problem for the model.

44. When2Call: When (not) to Call Tools

    When2Call evaluates whether a model should call a tool, request missing information, or acknowledge that available tools cannot answer. Its construction includes removing information needed for a required parameter and creating questions related to, but unsupported by, the supplied tool. The primary evaluation selects among behavioral alternatives rather than executing calls. This separates deciding whether a call is appropriate from checking a generated call’s arguments.

45. Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents

    Evaluate tool selection and argument correctness separately: choosing the intended function does not establish that the call is correct.

46. Function calling with the Gemini API

    The Interactions guide documents auto selection, which permits a direct response or function call; any, which requires a call; and none, which prohibits calls. Its allowed_tools example restricts eligible functions. The streaming example accumulates argument fragments separately by event index, retains each call's identity, and parses accumulated arguments after interaction.completed. The guide requires reconstructing complete calls before execution. It also explicitly documents combining function calling with structured output for Gemini 3 series models.

47. Let's Build an Agent from Scratch — Kam Lasater

    The refactored demo executes an array of tool calls concurrently and attaches a tool call ID to each returned result.

48. Function Calling is All You Need

    The demonstrated agent repeatedly processes model tool calls until the model returns no further calls.

49. Let's Build an Agent from Scratch — Kam Lasater

    In the demonstrated Chat Completions implementation, application code executes requested tools and appends both calls and results to the conversation before invoking the model again.

50. Vercel AI SDK Masterclass: From Fundamentals to Deep Research

    The weather example groups two independent retrieval calls into one step, then performs dependent arithmetic and final text generation in subsequent steps.

51. OWASP Authorization Cheat Sheet

    Authentication establishes identity, while authorization checks whether an entity may perform a requested action. OWASP recommends minimum necessary privileges, denial by default, and permission checks on every request regardless of its source. Its design guidance enumerates users, resources, and operations, including reads and writes, then tests that the resulting permission rules are enforced. Application implication: a corrected model proposal remains a new request requiring the same permission checks.

52. How we hacked YC Spring 2025 batch’s AI agents

    IDOR, or Insecure Direct Object Reference, occurs when tools accept object IDs without checking that the requesting user may access those objects.

53. OWASP Transaction Authorization Cheat Sheet

    Transaction approval should let users identify and acknowledge significant operation details. OWASP requires server-side enforcement, ordered authorization steps, protection against changing transaction data, and a final authorization gate tied to execution. Changed transaction details should invalidate prior authorization or restart the process. Authorization credentials should have a limited lifetime and be unique to an operation. The guide identifies approving one transaction but executing altered details as a time-of-check-to-time-of-use failure.

54. Your Agent Didn’t Fail. Your Harness Did.

    Approval must remain bound to one specific action and its scope, identity, arguments, and lifetime; expiration should terminate the approval path.

55. It's 10pm. Do You Know Where Your Agents Are?

    RFC 8693 Token Exchange is used to combine delegated user authority with runtime identity in a per-tool authorization request.

56. Function calling

    The application supplies tool descriptions and argument schemas; the model returns a proposed function name and arguments. The application parses arguments, dispatches its own code, and sends the resulting output back with the corresponding call identifier. The model can then answer or request further calls. Strict mode constrains arguments to the supported schema; object schemas require additionalProperties:false and required properties, with nullable types representing optional values. Application-side validation and authorization must occur before effects: a schema-valid customer identifier can still identify the wrong customer or an inaccessible account.

57. Tool use with Claude

    Claude’s documentation distinguishes application-defined tools, Anthropic-defined client tools, and server tools. The application executes its own tools and returns results. It also executes Anthropic-schema client tools, including bash and text editing, despite receiving their definitions from the provider. Server tools such as web search and web fetch execute on Anthropic’s infrastructure without application handler code. Definition ownership and execution ownership are therefore separate properties.

58. MCP tools: application security responsibilities

    MCP requires servers to validate tool inputs, enforce access controls, rate-limit invocations, and sanitize outputs. Clients should expose inputs before execution, seek confirmation for sensitive operations, validate results before passing them to the model, impose timeouts, and log usage. Input schemas describe admissible argument structure; they do not confer permission. The application must check the requested action and target against the caller's authority. When an output schema is provided, server results must conform and clients should validate them. Protocol errors and isError tool results must be handled as failures rather than successful actions.

59. AIP-151: Long-running operations

    Google's long-running-operation pattern returns an operation object instead of the ultimate result, allowing clients to track progress and retrieve the outcome. Metadata can report progress or partial failures. Errors that prevent starting are returned immediately; failures during execution belong in Operation.error. A resource being created or deleted may appear in reads while explicitly marked unusable.

60. RFC 9110: Conditional requests and successful response semantics

    If-Match conditions a request on the current representation's entity tag and uses strong comparison. The origin server evaluates it before applying the method; a false condition forbids performing that method, commonly producing 412. This prevents lost updates from concurrent writers. Separately, 202 means processing was accepted but remains incomplete and might never occur; its response should describe status and provide monitoring information. A 204 indicates successful fulfillment without response content. Application implication: preserve these distinctions when mapping provider responses into tool results.

61. GitHub REST API: Create an issue

    GitHub creates issues through POST /repos/{owner}/{repo}/issues with a required title. Fine-grained tokens require Issues write permission; disabled issues produce 410 Gone. The documented 201 response includes an issue id, repository-scoped number, title, and URLs. Creation can trigger notifications. Some requested attributes, including labels and assignees, are silently dropped when the user lacks push access. Consequently, even a creation response does not establish that every requested attribute was applied.

62. Your Agent Didn’t Fail. Your Harness Did.

    Bound external waits, record terminal outcomes, and keep recovery commands outside the blocked work queue.

63. Your Agent Is Wasting Tokens and You Don't Know It - Erik Hanchett, AWS

    Set a maximum iteration count for the agent's tool loop.

64. MCP Tasks extension: capability, lifecycle, and result retrieval

    Tasks are an optional extension, io.modelcontextprotocol/tasks, currently augmenting tools/call. Both peers declare support; clients include it in per-request capabilities. The server chooses an ordinary result or CreateTaskResult with resultType: task and taskId, and must durably create a discoverable task before returning its handle. Clients poll tasks/get, respecting pollIntervalMs. Working means execution continues; input_required exposes inputRequests answered through tasks/update. Completed includes the original method's result, even a tool result with isError; failed carries a JSON-RPC error; cancelled carries cancellation status. Optional notifications/tasks deliver equivalent full state and results through subscriptions/listen for acknowledged taskIds. Ordinary notifications/progress are unsupported for tasks. Retention uses creation time plus ttlMs; null means unlimited TTL, and TTL may change.

65. Instructor: Retry Logic with Tenacity

    Instructor documents selecting retryable exception types and imposing attempt or elapsed-time stop conditions. Failed attempts are passed to reask handlers for contextual correction feedback, and retry exhaustion exposes attempt history. Its runtime-context example checks that an extracted quotation occurs in supplied source text. This demonstrates a deterministic check beyond field types, followed by bounded regeneration when validation fails.

66. RFC 9457: Problem Details for HTTP APIs

    RFC 9457, published in July 2023, defines a machine-readable HTTP error object with a problem type, human-readable title and detail, advisory status, occurrence identifier, and type-specific extensions. It advises clients not to parse the human-readable detail for machine decisions and shows structured per-field validation errors using JSON Pointer locations. Problem details describe interface failures rather than exposing internal debugging data.

67. Resolving an ambiguous payment request

    A timeout can leave the client unable to tell whether Stripe received or executed a request. Stripe documents retrying with the same key and parameters until a server result is obtained, using backoff. An HTTP 500 remains indeterminate: side effects may exist even though the cached response stays unchanged. Stripe may reconcile partial mutations and emit webhook events for resulting objects. Supplying a local operation identifier in metadata lets the application correlate these objects with its own pending operation. Engineering consequence: preserve pending state until authoritative provider evidence resolves it; do not infer failure solely from a timeout.

68. Making Retries Safe with Idempotent APIs

    A timeout can leave the caller unsure whether a mutation succeeded. AWS describes caller-provided request identifiers reused for retries of the same intent, with duplicate detection scoped to the caller and identifier. Identical parameters alone do not establish identical intent. The service must durably coordinate recording the identifier with all related mutations as an ACID operation: recording first can suppress work that never happened, while mutating first can allow duplicate effects after a crash. Duplicate requests receive semantically equivalent responses; reusing a key with changed parameters produces a mismatch error. Retention must account for late retries, with a service-specific lifetime. Merely putting a key in logs supplies audit evidence, not atomic duplicate prevention. Application inference: a local key log cannot guarantee an external side effect is idempotent unless that effect participates in the relevant protocol.

69. Stripe retry keys and retention boundaries

    For a mutation, send an idempotency key and reuse that key with identical parameters when retrying an ambiguous connection failure. Stripe stores the first executed request's status and response body, including failures such as HTTP 500, and returns that result for repeated requests. Parameter mismatches produce an error. Keys may be pruned after they are at least 24 hours old; reuse after pruning creates a new request. Results are saved only after endpoint execution begins, so validation failures and conflicts with concurrently executing requests do not create a saved result.

70. Idempotent requests

    Stripe stores the first executing request's status code and response body for an idempotency key, including failures such as HTTP 500, and returns the stored result on retries. Parameter mismatches are rejected. A request rejected before execution because of validation or a conflict with a concurrently executing request does not create a saved idempotent result and can be retried. Keys may be pruned once at least 24 hours old; reuse after pruning starts a new request. Thus an in-progress conflict, a stored failure, and an expired key have different recovery semantics.

71. Temporal Activity Definition: Idempotency

    Idempotency means repeated execution has no additional externally visible effect beyond one execution. Completed activities are not rerun during workflow replay, but an activity can finish an external operation and crash before Temporal records completion; retry can repeat that operation. Temporal therefore recommends idempotent activities and explains that external services enforce idempotency keys. Application inference: use stable logical-operation keys for generation submissions, persist provider job identifiers, reconcile uncertain submissions before resubmitting, and deduplicate artifact registration. Deterministic workflow replay alone does not make provider calls exactly once.

72. LangGraph Functional API: deterministic resumption and idempotency

    Functional API resumption restarts the entrypoint and restores completed task and subgraph results from checkpoints. Ordinary entrypoint code runs again. Put randomness, clock reads, model calls and individual side effects inside tasks so persisted results can be reused. Keep task and interrupt ordering consistent with the recorded execution. A task that started but did not finish can execute again, including when an external effect occurred before its result was saved. Checkpointing therefore does not replace idempotency keys or checks for previously completed effects. Reusing a saved model result preserves that recorded value; calling the model again is a new inference, not reconstruction of an identical historical response.

73. BFCL V3: Multi-Turn and Multi-Step Function Calling

    BFCL V3 initializes each case with a backend fixture and evaluates every turn using both resulting state and required execution-path checks. State checks assess mutations, while required-call checks cover information requests that leave state unchanged. The authors explain that correct end state alone cannot establish that necessary information-gathering calls occurred. Their API fixtures also receive unit tests for individual functions, chained behavior, and error conditions. An entry must pass both checks in every turn; exceeding the documented step limit makes the entry incorrect.

74. Building enterprise LLM agents that work

    Build golden cases that specify expected function calls, parameters, tool outputs, and final responses, rather than evaluating only the final answer.

75. Pydantic AI: Unit testing

    Pydantic AI documents replacing a real model with TestModel or FunctionModel and inspecting captured request/result messages. TestModel procedurally generates schema-shaped arguments without understanding the task. In the published weather example, its chosen past date exercises historical lookup but misses the forecast branch; FunctionModel supplies controlled arguments to exercise that branch. Agent.override can replace models, dependencies, or toolsets. This provides a concrete way to test interface behavior separately from variable model selection.

76. Every Solo Agent Builder Eventually Reinvents a Worse Version of CI/CD

    The speaker identifies regression testing, CI monitoring, and contract testing as recurring operational needs in independently built agent systems.

77. Model Context Protocol: Architecture, revision 2026-07-28

    MCP separates a host application, its clients, and servers that expose capabilities. The host coordinates model integration, context aggregation, permissions, and consent; each client communicates with one server. Servers supply resources, tools, and prompts and may run locally or remotely. The 2026-07-28 architecture is explicitly stateless: every request carries protocol version and capabilities. Servers request client input through an InputRequiredResult in a reply. MCP standardizes communication; it does not supply the application’s planning policy or decide which actions are appropriate.

78. Model Context Protocol: Specification and trust principles, revision 2026-07-28

    MCP exposes resources for context and data, prompts for reusable messages and workflows, and tools for executable functions. These primitives have different jobs even when they concern the same underlying service. The specification requires user control over data access and operations, explicit consent before sharing data with servers, and careful handling of arbitrary tool execution. It also states that the protocol cannot itself enforce all trust principles: implementers need authorization flows, access controls, and clear interfaces. Interoperability does not eliminate the host’s security responsibilities.

79. Model Context Protocol: Tools, revision 2026-07-28

    Servers expose tools/list discovery and tools/call invocation. Definitions contain a name, description, input JSON Schema, and optional output schema; output-schema results must conform and clients should validate them. Protocol errors differ from execution failures returned with isError. Names are scoped to one server, so aggregation needs disambiguation. The available catalog can change over time or with request authorization, but not implicitly with connection state. Deterministic ordering is recommended. Servers advertising listChanged should notify clients subscribed through subscriptions/listen with toolsListChanged. Clients still consume pagination and caching rules rather than assuming the first list is permanently complete. Tool annotations are untrusted unless their server is trusted.

80. Remote MCPs: What we learned from shipping — John Welsh, Anthropic

    A shared integration interface reduces duplicated plumbing and makes integrations reusable across services; the speaker recommends MCP for presenting context to models.

81. Letting AI Interface with Your App with MCP

    Model Context Protocol (MCP) standardizes client-server communication while leaving service-specific capabilities with the service provider.

82. Model Context Protocol: Transport overview, revision 2026-07-28

    Transport bindings carry the same JSON-RPC protocol semantics through different channels. The stdio binding uses newline-delimited messages over a client-launched subprocess’s standard streams. Streamable HTTP sends messages by POST to one MCP endpoint, with replies as JSON or request-scoped SSE streams. Every request carries version and capabilities in its body metadata. The revision explicitly contrasts this with earlier connection-scoped initialize handshakes. Transport choice changes framing, deployment, and cancellation mechanics; it does not change a tool’s business meaning.

83. Pydantic is all you need

    Structured output needs schema validation and typed parsing, not merely a reliably located JSON string.

84. AI Engineering with the Google Gemini 2.5 Model Family

    MCP standardizes reusable tool integrations; the model still selects tools from declarations and the prompt, as in ordinary function calling.

85. Your Agent Didn’t Fail. Your Harness Did.

    Trace one real run from trigger identity through inherited state, authority, execution attempts, and surviving external evidence.

86. RFC 9396: OAuth 2.0 Rich Authorization Requests

    RFC 9396, published in May 2023, defines authorization_details for carrying fine-grained, API-specific authorization requirements rather than relying only on coarse scope strings. Authorization servers must reject unknown detail types, fields, wrong types, invalid values, or missing required fields. The RFC also warns that comparing two authorization-detail JSON objects mechanically is generally insufficient because field semantics can alter the rights granted.

87. Language Models (Mostly) Know What They Know

    Confidence calibration concerns agreement between predicted probabilities and observed correctness: among comparable predictions assigned probability p, approximately fraction p should be correct. The paper assesses this by binning predictions and plotting mean confidence against empirical correctness. For B equally populated bins its ECE is (1/B) sum_b |accuracy_b-confidence_b|. Its multiple-choice calibration charts include all answer options, whereas its ECE uses top predictions, so the evaluated population must be stated. The experiments find calibration depends on model, question format, and task; learned P(IK) struggles on new tasks. Application inference: emitting a confidence number in a typed record does not establish calibration. Compare those numbers with correctness labels on representative held-out examples using a specified scoring rule.

88. No more bad outputs with structured generation

    The speaker reports approximately negligible inference overhead for Outlines in a comparison with Guidance, while acknowledging a separate time trade-off.

89. Pydantic is all you need

    Put field descriptions, nested structures, and associated behavior into the model so the schema and prompting instructions share a reviewable definition.

90. Pydantic is all you need

    Separate validation, error handling, and re-asking, and feed validation failures back into a bounded retry process.