Observability

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. OpenTelemetry: Observability primer

   Observability supports investigating system behavior through emitted evidence. Telemetry is the data emitted about that behavior. Metrics aggregate numerical measurements over time; logs record timestamped messages; distributed traces connect operations across services. Logs become more useful for investigation when correlated with execution context. The primer distinguishes availability from meeting user expectations: a working shopping endpoint can still add the wrong item.

2. OpenTelemetry: Traces

   A distributed trace connects operations through spans carrying identity, timing, attributes and events. Parent-child relationships and links represent related work across boundaries. Applied to an agent, model requests, tool dispatch and downstream calls can be correlated so a failed run can be reconstructed from observations. A span’s error-free status describes that instrumented operation; it does not by itself establish that the requested business outcome was correct.

3. Google SRE: Effective Troubleshooting

   Begin with expected behavior, actual behavior and reproduction information. Combine telemetry with system knowledge to propose explanations, then seek confirming or disconfirming observations or make controlled changes. Inspect component interfaces and intermediate data to localize a failure; known test inputs and nonproduction reproduction can help. Preserve evidence while mitigating serious incidents. The chapter explicitly warns that correlated symptoms may share a cause or coincide, and that fixing a proximate problem can precede identifying its underlying causes.

4. OpenTelemetry GenAI model and tool spans

   GenAI spans describe logical operations through completion, error, or cancellation and can include automatic retries. Diagnostic attributes include provider, requested and returned model, prompt version, sampling parameters, token usage, finish reasons, and errors. Tool execution spans identify tool names and call IDs, enabling correlation with model requests. Instructions and messages should not be captured by default; instrumentation should offer opt-in. Tool arguments and results are explicitly opt-in and potentially sensitive. External content storage can separate content access controls from operational telemetry.

5. RabbitMQ: Consumer Acknowledgements and Publisher Confirms

   Publisher confirms concern the publisher's interaction with RabbitMQ and are unaware of consumers. Consumer acknowledgments concern broker-to-consumer delivery and processing; they are independent of publisher confirms. Automatic acknowledgment can consider delivery complete immediately after sending, whereas manual acknowledgment depends on the consuming application's decision. A delivery tag identifies a delivery only within its channel. Consequently, a broker confirmation cannot by itself establish that downstream application work completed.

6. Metrics design — vLLM

   Serving telemetry distinguishes queue time, prefill, decode, time to first token, inter-token latency and end-to-end latency. Event boundaries and observation location matter: client network time differs from engine intervals. A successful finish reason does not establish task correctness. Request lengths, waiting/running requests and KV usage help explain a latency change.

7. OpenAI Python SDK: ChatCompletionStreamOptionsParam

   With streaming include_usage enabled, an additional chunk before data: [DONE] contains aggregate token usage for the entire request and an empty choices array. Other chunks carry null usage. An interrupted stream may never deliver the final usage chunk. Ledger implication: when aggregate usage was not received, mark provider-reported request token totals unavailable rather than zero. Any task total depending on that missing attempt remains incomplete; keep independently obtained measurements or estimates explicitly separate.

8. LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize AI

   Runtime traces and spans provide an audit record of agent behavior that source code alone does not reveal.

9. Everything You Need To Know About Agent Observability

   The speakers distinguish explicit telemetry failures from less directly observable failures such as user frustration, positioning their system around the latter while retaining explicit error tracking.

10. How agent o11y differs from traditional o11y

    Agent observability adds contextual quality checks to technical telemetry, including grounding, tool selection, and brand alignment.

11. The NetLogger Methodology for High Performance Distributed Systems Performance Analysis

    Lawrence Berkeley National Laboratory’s NetLogger paper, published at HPDC in July 1998, addressed performance problems whose symptoms appeared far from their source. It combined application, operating-system and network events in a common format, recording when data was requested, received and processed. Its visualizations followed an object’s path through distributed components. Collection agents filtered and managed the resulting logs. The authors reported using this approach to diagnose intermittent stalls in their Distributed Parallel Storage System.

12. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

    Google’s April 2010 Dapper report described more than two years of building, deploying and using production tracing. To keep tracing continuously enabled with little application involvement, Dapper instrumented shared threading, control-flow and RPC libraries and used adaptive sampling. Its motivation was that aggregate request latency could reveal a problem without identifying the responsible subsystem.

13. Canopy: An End-to-End Performance Tracing And Analysis System

    Facebook’s October 2017 Canopy paper describes a March 2017 regression in which initial display of a particular page became approximately 300 ms slower. End-to-end traces showed elevated browser and resource-fetching time while server and network time remained relatively unchanged. Further breakdown identified a component adding 10 kB of CSS that disrupted early flush, the mechanism for sending predicted resources before they were requested. Engineers corrected the resource grouping. The investigation combined client observations, dependencies, component detail and historical comparisons.

14. X-Trace: A Pervasive Network Tracing Framework

    The 2007 X-Trace paper addressed diagnostic tools confined to one application or network layer. Its Berkeley researchers propagated a task identifier through recursive requests and lower protocol layers, connecting reports into a task tree. Implementations covered DNS resolution, a photo-hosting website and an overlay service. Administrative domains could retain their own reports while sharing the identifier for cooperative investigation. The paper explicitly distinguished showing the executed path from proving a fault’s underlying cause.

15. A brief history of OpenTelemetry (So Far)

    In their May 21, 2019 account, OpenTracing and OpenCensus leaders described merging the projects into OpenTelemetry. Both sought widely available instrumentation, but their similar, incompatible approaches created developer uncertainty and duplicated integration work. The merger prioritized consolidation, standardization and migration bridges rather than a single new tracing feature. Its initial governing and technical committees included representatives from Google, LightStep, Microsoft and Uber.

16. W3C Trace Context Recommendation

    The W3C Recommendation dated November 23, 2021 standardizes HTTP propagation for distributed tracing. traceparent carries a portable trace identifier, current parent identifier, flags, and version; tracestate carries optional vendor-specific context. A receiver updates parent identity for downstream requests. Invalid context can cause a new trace to begin. The sampled flag communicates a recording preference, but the specification says it does not guarantee that telemetry will be recorded.

17. Sunsetting OpenCensus

    OpenTelemetry’s May 1, 2023 announcement reported feature parity with OpenCensus across several languages and scheduled most OpenCensus repositories for archival on July 31, 2023. Compatibility bridges supported incremental migration. This later announcement shows that the 2019 merger announcement and implementation transition were separate milestones.

18. The LinkedIn Generative AI Application Tech Stack: Extending to Build AI agents

    LinkedIn’s September 10, 2025 account describes different capture choices for development and production. Pre-production used LangSmith to inspect model calls, tools and control flow with rich execution context. Production used leaner OpenTelemetry spans for model calls, tool invocation and memory usage, connecting agent behavior to upstream requests and downstream services. Persisted traces also supplied datasets for offline evaluation and regression testing. The workflow illustrates extending distributed tracing with AI-specific boundaries while changing capture detail for production constraints.

19. OpenTelemetry GenAI agent spans

    Agent invocation spans identify the invoked agent with its available name, stable ID, and version, alongside operation, provider, conversation identifier, and error information. Distinguishing agent identity and version from individual model calls makes it possible to associate failures with the orchestration configuration that invoked them.

20. Traces — OpenTelemetry

    A span records an operation with start and end times, an identity, and optional attributes, events, status, and links. Parent-child relationships describe nested work within a trace. Events mark meaningful instants rather than durations, while attributes attach metadata. Links can associate asynchronously started work with an earlier operation even when the later operation belongs to a separate trace. These distinctions support reconstructing request execution without forcing a queued job or concurrent branch into a misleading linear log.

21. OpenTelemetry: Semantic conventions for messaging spans

    Messaging conventions distinguish message creation, sending, receiving, processing and settlement. Receive and process spans should link to the creation context of each message they account for. Links are the default producer-consumer correlation mechanism because batches can have multiple originating messages while a span has only one parent, and consumption may occur inside another active operation. For single-message processing, using the creation context as parent is permitted under documented conditions. Per-message attributes that differ within a batch belong on the corresponding links.

22. OpenTelemetry: Context propagation

    Context propagation connects work across service boundaries by carrying trace and parent-span identity to the receiving process. With W3C Trace Context, the sender injects traceparent and the receiver extracts it so downstream spans belong to the same request. Trace and span identifiers can also correlate logs with that execution. Without propagation, individually instrumented services can still produce disconnected records. Incoming trace context is not inherently trustworthy, and propagated baggage can leak sensitive information; correlation metadata needs an explicit trust boundary too.

23. Temporal: Workflow Id and Run Id

    Temporal distinguishes an application-level Workflow Id from the platform-level Run Id of an individual execution. Workflow retries and operations such as Continue-As-New create new executions and change the current Run Id. Namespace, Workflow Id and Run Id together identify an execution. Workflow identifiers are visible in operational interfaces and logs and are not protected by payload codecs, so they should not contain sensitive data.

24. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

    Dapper records timestamped span events alongside trace, span and parent identifiers used to reconstruct execution relationships. A single RPC span commonly contains observations from both client and server hosts. Because their clocks can disagree, its analysis tools use known communication ordering: the client sends before the server receives, and the server sends its response before the client receives it. These relationships bound server-side timestamps. Diagnostic implication: timestamps from different hosts alone cannot establish precise event ordering or one-way latency; preserve explicit communication and parent relationships rather than sorting all records as though their clocks were synchronized.

25. OpenLLMetry is all you need

    Logs capture individual events, metrics describe aggregate behavior, and traces follow multi-step execution.

26. OpenTelemetry Logging Specification

    OpenTelemetry correlates logs by execution context, origin, and time. Log records can carry trace and span identifiers so records from participating components can be joined to the same execution. Resource context identifies the service or infrastructure that emitted the record. The specification supports translating existing log formats into a common data model while providing structured events for new instrumentation. Logs, metrics, and traces therefore remain different records whose shared context enables navigation between them.

27. OpenTelemetry Metrics API: Gauge

    An OpenTelemetry Gauge records a current absolute, non-additive numeric value rather than an increment or accumulated change. The synchronous Gauge is intended for values delivered through change-event subscriptions. When a current value is obtained through an accessor, the API directs implementations toward an asynchronous ObservableGauge callback. Values that are meaningfully additive across sources belong in an UpDownCounter instead.

28. OpenTelemetry Metrics Data Model

    OpenTelemetry transforms individual metric observations into streams and time series because exporting every raw observation can be infeasible in network and CPU terms. A time series is identified by metric name, attributes, value type, and unit. Temporal reaggregation lowers time resolution; spatial reaggregation removes attributes. An exemplar retains a selected observation with its value, time, optional trace and span identifiers, and filtered attributes, allowing an aggregate metric point to lead investigators to a particular execution.

29. Histograms and summaries — Prometheus

    Histograms preserve bucketed observations, their count and sum; summaries can expose precomputed quantiles. Counters increase except at resets, and rates describe their change over a chosen interval. Compatible histograms can be aggregated before estimating a fleet percentile; averaging instance percentiles does not produce that percentile. A constructed serial-latency example also shows why component percentiles cannot be added: among 100 requests, stage A takes 10 ms on four requests and 1 ms otherwise; stage B does likewise on four different requests. Each stage’s nearest-rank p95 is 1 ms, but total p95 is 11 ms, not 2 ms.

30. Prometheus: metric-label cardinality and resource costs

    For a metric, each distinct label set identifies another time series with RAM, CPU, disk and network costs. If all combinations occur, label dimensions with c_1,...,c_k values can produce product_i c_i series, multiplied further by targets and metric components. Adding request IDs, user IDs or arbitrary prompt text can therefore grow series counts with traffic rather than with a bounded operational taxonomy. Use bounded labels for aggregate questions. Put request-level identifiers in appropriately protected structured logs or span attributes and correlate them with traces instead of making each request a metric dimension.

31. OpenTelemetry: Handling sensitive data

    Telemetry instrumentation cannot determine on its own which fields are sensitive in a particular application. OpenTelemetry recommends collecting only attributes that serve an observability purpose, reviewing instrumentation output, and considering aggregates or anonymized data. Collector processors can remove attributes, filter whole records, transform values, or enforce an attribute allowlist. This makes observability a data-design decision: retain enough context to answer a specific diagnostic question, while recognizing that full prompts, tool results, tokens, and identifiers can create an additional sensitive dataset.

32. OpenTelemetry: Sampling

    Head sampling decides before the full trace is known, making it efficient but unable by itself to retain every trace that later fails. Tail sampling waits for all or most spans and can retain traces with errors, high latency, or selected attributes, at the cost of buffering, state, and operational complexity. Combining them cannot recover traces discarded upstream. A diagnostic sample enriched for failures is therefore not a representative estimate of traffic quality; rates need a known denominator and a sampling design appropriate to the question.

33. Transformers: chat templates serialize conversation history

    A chat model still receives a token sequence. A chat template turns role-labelled messages into that sequence, adding model-specific control tokens that mark speakers and message boundaries. Retained user messages, assistant messages and instructions therefore consume input tokens alongside the latest question. The rendered template, including any assistant-generation prefix, determines the effective input. Counting only visible message text can miss control-token overhead. Stored conversation history is not automatically identical to the history actually serialized for a particular inference call.

34. Transformers GenerationConfig: stopping and output budgets

    Generation can end on an end-of-sequence token, a configured stop string, a length limit, or other stopping criteria. max_new_tokens limits generated tokens independently of prompt length; max_length includes the prompt. These are upper bounds, not promises to produce that many tokens. max_time is not a hard deadline: generation finishes its current pass after the allotted time. For ordinary single-beam generation, do_sample selects sampling versus greedy decoding. An application should distinguish normal completion from length truncation, cancellation and failure rather than interpreting every returned prefix as a completed answer.

35. OpenTelemetry GenAI semantic conventions: spans

    gen_ai.request.model records the requested model; gen_ai.response.model records the actual model when available. gen_ai.request.* includes temperature, top_p, top_k, seed, max_tokens and stop_sequences. gen_ai.conversation.id correlates an existing conversation; do not invent one from a trace ID. Opt-in gen_ai.input.messages records ordered input history, gen_ai.system_instructions separately supplied instructions, and gen_ai.output.messages responses. Tool execution uses gen_ai.tool.call.id, arguments and result. Content may instead be stored externally; reference representation remains application-defined. gen_ai.prompt.name/version support prompt provenance. Code revision, policy revision, retrieval corpus/index version and workflow definition/run/attempt identity still require application instrumentation and explicit conventions; invoke_workflow alone is insufficient.

36. Your Agent Failed in Prod. Good Luck Reproducing It.

    Capture inputs and outputs at workflow-node boundaries, including local operations, rather than relying exclusively on network capture.

37. Your Agent Failed in Prod. Good Luck Reproducing It.

    Inspecting adjacent node inputs and outputs can distinguish a tool executing bad arguments from an upstream model generating those arguments.

38. Node.js HTTP: server response completion and termination

    ServerResponse finish means the final response headers and body were handed to the operating system for transmission; it does not establish client receipt. close can indicate either completed output or premature connection termination. Inspect writableFinished to distinguish completed flushing from closure before that boundary; writableEnded only records that end() was called. An error event records an emitted error, not client consumption; outgoingMessage.destroy(error) can explicitly supply one. Application-design implication: record server flushing separately from client acknowledgment. If the example claims client consumption, define an acknowledgment sent after the client parses or otherwise processes the identified final response, and correlate it with the task and response identity.

39. Making Retries Safe with Idempotent APIs

    A timeout can leave the caller unsure whether a mutation succeeded. AWS describes caller-provided request identifiers reused for retries of the same intent, with duplicate detection scoped to the caller and identifier. Identical parameters alone do not establish identical intent. The service must durably coordinate recording the identifier with all related mutations as an ACID operation: recording first can suppress work that never happened, while mutating first can allow duplicate effects after a crash. Duplicate requests receive semantically equivalent responses; reusing a key with changed parameters produces a mismatch error. Retention must account for late retries, with a service-specific lifetime. Merely putting a key in logs supplies audit evidence, not atomic duplicate prevention. Application inference: a local key log cannot guarantee an external side effect is idempotent unless that effect participates in the relevant protocol.

40. What We Learned From A Year of Building With LLMs

    Associate traces with code, model, and prompt versions, and pin API model versions to reduce unexplained behavioral changes.

41. OpenTelemetry Telemetry Schemas

    OpenTelemetry schema URLs identify retrievable, versioned schema files. Published schema files are immutable, and OTLP can attach schema_url to resource and instrumentation-scope groups containing spans, events, metrics, or logs. A schema-aware consumer can translate a recorded attribute name from one declared version to another. Recording instrumentation scope and schema identity therefore helps distinguish a changed field definition from a changed system behavior when comparing historical telemetry.

42. The State of MCP Observability: Observable.tools — Alex Volkov and Benjamin Eckel, Weights & Biases and Dylibso

    An externally operated MCP server can remain a single opaque span, while a controlled server can contribute internal spans to the client's trace.

43. The State of MCP Observability: Observable.tools — Alex Volkov and Benjamin Eckel, Weights & Biases and Dylibso

    The working example carried trace context through MCP's meta payload and restored it on the server to preserve the parent relationship.

44. OpenTelemetry Baggage API

    OpenTelemetry Baggage is an immutable set of application-defined name/value properties associated with a distributed request or workflow. It is separate from trace identity and becomes a span, metric, or log attribute only when instrumentation explicitly copies it. Baggage may cross arbitrary process boundaries, so the API must permit clearing every entry before calling an untrusted process. This supports carrying bounded business correlation context without treating baggage as part of the trace protocol itself.

45. OpenTelemetry: Tracing SDK

    The standard simple and batching processors export finished spans. The batching processor has a bounded queue and drops spans when its queue limit is reached. Span processor callbacks run synchronously and should not block or throw. ForceFlush and shutdown can fail or time out; honoring a timeout can require skipping or aborting export work. A flush request therefore does not establish complete delivery.

46. Resiliency — OpenTelemetry Collector

    Collector sending queues buffer telemetry while an export destination is unavailable, and retries use backoff and jitter. A full queue or exhausted retry duration can still drop data. A persistent write-ahead log can preserve queued records across Collector restarts, but disk failure, capacity limits, or prolonged outages remain loss paths. Queue occupancy, queue capacity, and export-failure metrics therefore belong in the observability system itself. A quiet dashboard can reflect a broken evidence pipeline rather than healthy application behavior.

47. OpenTelemetry Collector Contrib: Tail Sampling Processor

    The tail-sampling processor requires a trace’s spans to reach the same Collector instance. Its bounded buffer can evict a trace before a sampling decision. Spans arriving after a decision inherit it while it remains available; after eviction, an uncached trace can be treated as new and receive a different decision. A decision cache preserves earlier choices only while entries remain cached. The processor exposes early-drop and late-arrival measurements and can attach sampling-policy metadata.

48. OpenTelemetry: OTLP Specification 1.11.0

    OTLP documents that a sender may not know whether telemetry arrived when its acknowledgment is lost. Resending after a reconnection or network interruption can therefore create duplicate data at the receiver. Duplicate delivery is an explicit protocol tradeoff, rather than proof that the application performed the represented operation twice.

49. LangGraph: checkpoint state and execution identity

    A checkpointer stores graph state at super-step boundaries. thread_id identifies the continuing thread; checkpoint_id identifies a particular saved snapshot, with checkpoint_ns distinguishing checkpoint namespaces. StateSnapshot includes channel values, next nodes, configuration, metadata, parent checkpoint configuration and pending tasks. Resumption restores persisted state, not an arbitrary process stack or every local variable. Successfully persisted work can survive a later failure; survival across process loss requires an appropriate persistent checkpointer, not merely an in-memory example. Application request, attempt and trace identities should be explicitly linked to the thread and checkpoint rather than treated as interchangeable.

50. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

    The study changes irrelevant or biasing features in model inputs, such as the ordering of multiple-choice options, and observes that explanations can omit those influences while rationalizing the resulting answer. A plausible written rationale can therefore be an unreliable account of what caused a prediction. For application diagnosis, recorded inputs, actions, and outcomes establish what was observed; interventions such as controlled input substitutions are needed to test a proposed causal explanation.

51. Anthropic: Reasoning models don't always say what they think

    Anthropic tested whether models acknowledged answer hints that influenced their responses. In the tested settings, Claude 3.7 Sonnet and DeepSeek R1 often adopted hints without mentioning them in their generated reasoning. Separate reward-hacking experiments produced explanations that omitted exploited hints or rationalized incorrect answers. These observations show that a generated reasoning account need not faithfully disclose factors affecting an answer.

52. LangGraph: Use time-travel

    LangGraph's time-travel replay resumes from a selected checkpoint by re-executing subsequent nodes. Model calls, API requests and interrupts can run again and produce different results. Forking uses an updated state to create a new checkpoint branch while preserving the original history. Checkpoint placement determines how finely execution can be revisited; a subgraph without its own checkpoint history is re-executed as a unit.

53. LangGraph: replaying from an earlier checkpoint

    Time-travel replay starts from a selected historical checkpoint. Nodes before that checkpoint are skipped because their results are saved; nodes after it execute again, including LLM calls, API requests and interrupts. Replaying a final checkpoint with no next nodes does nothing. Forking changes state into a new checkpoint while retaining the original history. This differs from fault resumption that reuses completed task results: historical replay intentionally re-executes subsequent work and may produce different outcomes. Reproducing an old answer exactly requires retaining and reading the original response, not assuming a new call will regenerate it.

54. Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

    Machine-using agents accumulate valuable filesystem, memory, and process state that the speaker proposes preserving separately from context.

55. Mind the Gap (In your Agent Observability)

    Attach evaluation results to their corresponding traces so a failed metric can be followed directly to the execution step and compared across versions.

56. Lawrence Jones - Fighting AI with AI

    A reliable prompt-editing loop still requires locating the faulty component in a larger hierarchy of agents, prompts, and tools.

57. VCR.py: Usage and record modes

    VCR.py can replay recorded HTTP interactions. Its none mode replays existing interactions and raises an error for new requests, making it suitable for tests where fresh HTTP calls would be dangerous. The default once mode can record new interactions when no recording exists; new_episodes permits additional recordings; all makes fresh requests instead of replaying recorded responses. Replay safety therefore depends on selecting the intended mode explicitly.

58. OpenTelemetry GenAI semantic conventions: metrics

    The current GenAI metrics distinguish client operation duration, token usage, and streaming time to the first response chunk. Client chunk timing is not interchangeable with the server's time-to-first-token metric: they have different observation points, and chunks need not correspond one-for-one to tokens. Token usage should be reported when it can be obtained efficiently, using provider usage when available; absent usage must not be fabricated. Model, provider, and operation dimensions help separate workload changes from changes in performance rather than collapsing all calls into one average.

59. CRISP: Critical Path Analysis of Large-Scale Microservice Architectures

    CRISP’s 2022 paper defines the critical path as the longest weighted dependency path from start to finish; total work instead sums all task weights. Inclusive RPC time spans its start to finish, while exclusive time excludes callees. CRISP computes paths per request because different requests can have different bottlenecks. For a constructed parent lasting 100 ms with child intervals [10,70] and [40,90], their union covers 80 ms, leaving 20 ms outside child intervals; subtracting both durations separately incorrectly gives −10 ms.

60. AI Engineering 201: The Rest of the Owl

    Monitor extreme latency quantiles and throughput separately, with diagnostics matched to the inference deployment model.

61. Designing Voice Agents for Real Conversations

    Evaluate tail latency as well as median time to first token because occasional slow responses can break conversational flow.

62. LLM cost management: how to track, attribute, and control spend — Langfuse

    Langfuse documents a concrete double-counting failure: application instrumentation and gateway callbacks can report the same model request as two generations, doubling aggregated cost. Its guidance calls for one accounting path per request, mutually exclusive usage buckets and the price applicable when the call ran. Application identifiers enable attribution to sessions, features and environments. It also supports separately recording metered tool costs. Inferred costs are computed at ingestion; correcting a price definition affects new generations rather than automatically repairing historical records.

63. Agentic SDLC at Uber - Building Blocks for Uber’s Software Factory

    Uber places request attribution, policy middleware, and audit-session capture in a shared Model Gateway.

64. OpenTelemetry Metrics — gRPC

    gRPC separately measures logical client calls and their attempts because retries or hedging can create multiple attempts. grpc.client.call.duration measures completion time from the application’s perspective. grpc.client.attempt.started counts attempts, including unfinished ones, while attempt.duration records completed-attempt durations. Experimental retry instruments record retry counts and time during which a call has no active attempt. Constructed example: one call that fails once and then succeeds produces two attempts but only one logical call.

65. How do I export monthly usage details from the API Usage Dashboard?

    OpenAI distinguishes activity exports from cost exports. Its reconciliation guidance uses cost data grouped by invoice line item, with the same organization and calendar-month service period as the invoice. Dashboard dates use UTC. Activity data can instead be grouped by dimensions including project, model, API key, batch and service tier. These are different views of usage and expense, rather than interchangeable per-request ledgers.

66. Allocation — FinOps Framework

    FinOps distinguishes directly assignable expenses from shared costs such as common platforms, networking and support. Shared costs require an explicit allocation strategy: they may remain centrally funded or be distributed using fixed proportions, usage proportions or proxy measurements. Account structures and metadata identify recipients; utilization and observability data can support finer allocation. Thus a workflow’s measured model-call expense and its assigned share of infrastructure answer different accounting questions.

67. How Forward Deployed Engineering is done at Cognition

    The speaker contrasts 'token maxing' with 'intelligent orchestration': deployment success should depend on measurable delivery outcomes rather than consumption.

68. Implementing SLOs — Google SRE Workbook

    A service-level indicator measures a defined aspect of service, often as good events divided by eligible events. A service-level objective sets a target for that indicator over a specified period; the remaining allowed fraction of bad events is the error budget. The indicator must reflect the customer experience, and the organization needs an explicit policy for how budget consumption changes decisions. Measuring a number without defining its population, success criterion, period, and owner does not create a useful reliability contract.

69. Google SRE: Monitoring Distributed Systems

    The four golden signals are latency, traffic, errors and saturation. Traffic should reflect the service's workload; saturation measures pressure on constrained resources. Track successful and failed request latency separately, retaining visibility into slow failures. Errors include incorrect content returned with HTTP 200 and violations of an explicit service policy, not merely protocol errors. Latency distributions expose slow requests hidden by averages. User-visible monitoring and internal measurements answer complementary questions; alerts should call for meaningful investigation or action.

70. Production Evals For Agentic AI Systems

    Apply an SRE or production-engineering lens: assess delivered value, operational reliability, human burden, risk, user experience, scalability, and resilience.

71. AI System Design: From Idea to Production

    Continue tracking offline metrics in production and add human override frequency and review duration as investigation signals.

72. LangSmith: Feedback data format

    LangSmith stores feedback separately from execution records and associates it with a trace or intermediate run. Sources include application feedback, human annotation and offline or online evaluators. The documented record includes run identity, creation and modification timestamps, a criterion key, numerical or categorical judgment, comments, correction information and feedback-source details. Source metadata and user identity can accompany the judgment.

73. National Academies: inference with missing outcomes

    If O indicates an observed outcome and Y indicates success, the observed rate estimates P(Y=1|O=1), not automatically P(Y=1). Selective feedback can make these differ. Missing completely at random means missingness is independent of measured and missing values; missing at random permits dependence on observed information but not remaining missing outcomes after conditioning. MAR therefore does not justify an unadjusted complete-case success rate. Adjustment or weighting needs justified models and adequate observation probabilities. Dependence on unobserved outcomes requires additional assumptions and sensitivity analysis. Derived bookkeeping: with S successes, F failures and U unresolved eligible cases, the eventual success fraction lies between S/N and (S+U)/N, where N=S+F+U.

74. National Academies: informative censoring and sensitivity analysis

    An outcome can remain unobserved because follow-up ends before the event, or because the subject disappears from observation. Censoring can be informative when the reason observation ends is related to the event time. A scheduled cutoff and dropout need not have the same missingness mechanism. Time-to-event inference requires justified assumptions about event and censoring dependence, potentially conditional on recorded prognostic variables; sensitivity analysis explores remaining dependence. Applied to AI workflows, recent requests have had less time to receive downstream outcomes. Compare cohorts at a specified follow-up horizon, report unresolved counts, and avoid counting absence of a recorded failure as success.

75. Operationalizing Machine Learning: An Interview Study

    Interview participants reported that ground-truth feedback for live predictions could arrive late, unpredictably or not at all, limiting knowledge of current production performance. Reported operational problems also included missing, incomplete or corrupted data. The study describes difficulties choosing useful data alerts: simple checks can miss errors, while distribution-distance alerts can produce excessive false positives.

76. Understanding Simpson’s Paradox — Judea Pearl

    For variant v and disjoint, exhaustive group g, define integer successes s_vg and totals n_vg>0, with 0<=s_vg<=n_vg. The aggregate rate is R_v=sum_g s_vg/sum_g n_vg=sum_g w_vg r_vg. A strict reversal requires r_Ag>r_Bg in every group but R_A<R_B. Different weights are necessary; identical weights preserve the within-group ordering. Derived count example: easy tasks A=9/10 and B=80/100; hard tasks A=20/100 and B=1/10. A wins within both groups, yet totals are A=29/110 and B=81/110. Here A receives many more hard tasks. A changed task mix alone can move an aggregate without any reversal; reversal additionally requires opposite aggregate and within-group comparisons.

77. Practical Guide to Controlled Experiments on the Web

    Concurrent randomized treatment and control groups support causal comparison when assignment produces comparable populations, users receive consistent variants, and interactions or interference do not undermine the design. Choose the evaluation criterion before inspecting results. Define eligible users and actual exposure; unexposed users can dilute effects, but exposure filtering must preserve comparability. Statistical power is the probability of detecting a specified real effect under the design. For independent observations, standard error decreases approximately as 1/sqrt(n); detecting smaller effects requires substantially more data. A migration test that finds no significant difference can still miss a harmful change when power is low.

78. OpenTelemetry Tracing SDK: sampling

    TraceIdRatioBased deterministically selects a configured ratio using the trace ID, but ignores the parent Sampled flag. ParentBased delegates root decisions to a configured sampler and, by default, records sampled children and drops unsampled children for both local and remote parents. Propagating SpanContext therefore lets downstream ParentBased samplers follow the upstream decision, keeping one distributed trace consistently sampled when propagation and configuration are intact.

79. OpenTelemetry Collector Contrib: Tail Sampling Processor

    The Collector tail sampler supports probabilistic selection; token-bucket rate limiting; numeric, string, boolean, trace-state, trace-flag, and OTTL attribute conditions; status-code selection; elapsed-trace latency thresholds; and ordered composite policies with rate allocation. A known probabilistic policy can support adjusted population estimates under its assumptions. Attribute, status, and latency policies describe deliberately selected strata, not their prevalence in all traffic. Rate-limited output is a capacity-bounded retained set, not automatically representative. Composite output inherits the selection properties of its constituent policies and has no single unbiased population interpretation without recording those policies and inclusion behavior.

80. An Online Probabilistic Distributed Tracing System

    Toslali and colleagues' May 2024 Astraea paper separates tracing costs into application runtime work to create spans, network work to route them, and backend storage and computation. In experiments on three microservice applications with injected performance variations, Astraea localized the responsible region 92% of the time while enabling 25% of available instrumentation; reported case studies used 11–28%. The system learns per-span sampling probabilities, trading exploration of changing problems against lower telemetry volume.

81. OpenTelemetry: Handling sensitive data

    OpenTelemetry cannot determine what is sensitive in an application's context; implementers must review emitted telemetry and protect it. The guidance favors collecting only data needed for observability and avoiding unnecessary personal information. Collector processors can delete or modify attributes, filter entire records, enforce attribute allowlists, and transform values. Credentials and session tokens are among the explicitly identified sensitive categories. Hashing predictable identifiers does not reliably anonymize them because candidate values can be enumerated.

82. OWASP: Logging Cheat Sheet

    Avoid directly logging passwords, access tokens, session identifiers, encryption keys, database connection strings, payment details, sensitive personal data and commercially sensitive content. Remove or appropriately transform sensitive fields before recording events. Restrict and periodically review log access, monitor access, protect integrity and secure transmission. Retention rules must cover debug logs, backups, copies and exports as well as primary storage. Engineering application: prompts, retrieved passages, tool arguments and model outputs require the same classification and redaction controls as ordinary application data.

83. OpenLLMetry is all you need

    A self-deployed OpenTelemetry collector can preprocess telemetry before it reaches an observability provider.

84. OpenTelemetry GenAI client spans and content capture

    The development conventions treat instructions, user messages and model outputs as sensitive and recommend not capturing them by default. Tool arguments and results are opt-in attributes and may contain sensitive data. Content can be omitted, attached to spans, or stored externally with references and separate access controls. Structured content may be truncated. A configured external-storage hook operates independently of content opt-in flags and should run regardless of span sampling, so disabling span content or sampling alone does not necessarily stop content collection.

85. NIST Privacy Framework 1.0: lifecycle and minimized audit evidence

    The framework inventories data elements, processing purposes, actions, owners and flows. Policies define permitted uses and retention periods; the data lifecycle aligns with system development and operations. Authorizations must be maintained and revocable, access limited by least privilege, and deletion and destruction performed under policy. Audit records themselves must incorporate data minimization. Engineering application: define the decision evidence needed for review, its purpose, authorized readers, retention trigger and disposal method before logging. Retain the necessary decision, model and policy versions and relevant evidence without indiscriminately copying personal data into logs, prompts or backups. Where review requires sensitive evidence, constrain fields, access and retention rather than treating auditability as permission to keep everything. Assess removal and disclosure across downstream copies and service providers.

86. How agent o11y differs from traditional o11y

    Agent traces combine semi-structured records with substantial unstructured text, creating ingestion, processing, and inspection challenges.

87. Why building eval platforms is hard

    Agent trace storage must handle the combination of large, text-heavy, weakly structured records, fast ingestion, and heterogeneous reads.

88. Billable Units — Langfuse

    Langfuse defines telemetry units as the sum of ingested traces, observations and scores. Records created by evaluation or annotation features also count. Under that documented contract, a constructed execution containing one trace, six observations and two scores contributes nine units, not one. This demonstrates why additional instrumentation and assessment can increase the observability bill independently of model-token spending.

89. Configure Tempo — attribute limits and compression

    Tempo documents memory problems when querying traces with unusually large attributes. Its max_attribute_bytes setting truncates oversized attribute keys or values before storage, with a counter and diagnostic logs exposing truncation. The documentation also describes a compression tradeoff: disabling compression can reduce CPU and memory work while increasing network traffic and live-store volume. Payload capture and transport settings therefore change both diagnostic fidelity and operational resource use.

90. Your Agent Failed in Prod. Good Luck Reproducing It.

    Record execution inputs and outputs together with session variables such as model version, code version, and retrieved content.

91. Using RL-based Agent to Detect and Remediate ETL Pipeline Failures

    An event-driven recovery loop combines read-only diagnostic evidence, guarded execution, and validation after rerunning the job.

92. AWS Builders' Library: timeouts, retries, backoff and jitter

    Retries add work precisely when an overloaded dependency may have least spare capacity. Nested retry layers multiply attempts: if layer i permits a_i total attempts, one operation can cause up to product_i a_i deepest calls when every layer exhausts its policy. Three attempts at each of five layers permit 243 calls. Serial attempts and backoff also increase elapsed latency; a caller timeout need not stop downstream work. Bound attempts, choose one appropriate retry layer, use timeouts, capped backoff and jitter, and constrain retry volume. A timeout means the caller stopped waiting, not that an external side effect failed to occur. Idempotent operations permit recovery without duplicating the intended effect.