Contents

Purpose and workload
Workload economics
1. Price accepted outcomes
  1. A cheaper call, a dearer result
2. Separate savings from allocation
Time and consequential delay
1. Budget the completion path
2. Find the delay worth removing
Avoided work and valid reuse
Usable capacity
Optimization experiments
1. Test the complete workload
2. Compare interventions across layers
Adoption and maintenance
1. Choose and revisit the operating point
Check understanding
Open questions
Selected talks
References
Talk library

AI Cost and Performance Engineering

AI cost and performance engineering reduces the resources needed to deliver acceptable results within the time users can wait. It examines the complete workflow: model calls, tools, verification, retries, human review, and the capacity kept ready to serve them. The goal is to remove unnecessary work and consequential delays while preserving quality and operating requirements. A cheaper call matters only if the resulting workflow remains useful and economical.

Purpose and workload

Optimize acceptable work

Start by defining acceptance criteria: observable conditions that make a completed task useful. A generated database query may need to return the intended records, respect access restrictions, and finish while someone is waiting. Producing valid query syntax meets only part of that contract. An evaluation systematically assesses behavior against its intended use; defining worthwhile improvement establishes the baseline and checks that this chapter takes as inputs.

Separate a constraint, which a candidate must satisfy, from an objective, which you improve among acceptable candidates. Quality, completion deadlines, failure rates, and review burden can constrain a cost-reduction study. Interactive completion and background processing need different operating points because the user waits differently. Hacking the Inference Pareto Frontier develops this application-first framing: establish the required quality and latency before minimizing cost.

A service-level indicator, or SLI, defines a quantitative observation; a service-level objective, or SLO, gives it a target over a specified population and window. A response-time target needs start and end events, an observation location, and an eligible population. Receiving an acknowledgement, seeing useful partial output, and receiving a completed usable result are different boundaries. A system can improve one while leaving another unchanged.

Foundational developments

Whole-workload optimization brings together several older engineering traditions. Each addresses a different source of expense or delay; their coexistence explains why improving model execution alone rarely settles the complete decision.

Critical-path scheduling — 1959 — Kelley and Walker's Critical-Path Planning and Scheduling represented activities and dependencies as a network. Completion depended on the longest required path, making the placement of work as consequential as its duration.
Amdahl's argument — 1967 — Gene Amdahl's whole-problem analysis challenged acceleration claims that neglected sequential data management and irregular work. Unchanged portions of a computation constrain the benefit of faster arithmetic.
The working-set model — May 1968 — Peter Denning's working-set paper connected scheduling with recently referenced information. Too little retained memory caused repeated retrieval; too much consumed capacity that other work needed.
Cloud elasticity — February 2009 — Armbrust and colleagues' Berkeley report, Above the Clouds, explained how acquiring and releasing resources with demand could reduce both idle provisioning and insufficient capacity. Hourly price alone could not describe workload economics.
Joint agent optimization — 2024 — Kapoor and colleagues' AI Agents That Matter separated one-time optimization expense from recurring execution cost and studied quality and cost together. Deployment volume therefore became part of judging an optimization's value.

These developments are complementary, not a replacement sequence. Dependency analysis identifies consequential waiting; memory analysis explains reuse; elasticity changes what capacity must be purchased; task-level optimization asks whether the resulting work remains worthwhile.

Describe the demand

Use three nested units. A task is the requested outcome. An attempt is one execution trying to achieve it. A call invokes a model, tool, or service within an attempt. One task may require several attempts, and each attempt may contain many calls. Preserve these relationships in the execution records used for analysis; counting API calls alone cannot describe task volume or expense.

Tokens are model-specific units used to encode inputs and outputs; count the complete request, including its formatting and supporting context. Keep input and output counts separate. Prefill processes supplied input, whose tokens are already known and can be processed together. Decode incrementally produces output: each new token depends on the preceding choices. Swapping a long output for an equally long input therefore changes execution demand even when total token count stays constant. Inference Engineering explains those mechanics.

Build a profile for each meaningful task class, then compare its share of task volume, model work, and total expense.
Profile dimension	Keep visible
Execution	Attempts per task; model and tool calls per attempt; failures and cancellations
Size	Input and output distributions, not only their averages
Timing	Arrival bursts, pauses between related calls, and completion deadlines
Reuse	Repeated inputs, compatible prefixes, and actual reuse outcomes
Outcome	Acceptance, review effort, and unresolved assessment

Observed traffic can depart sharply from an average request model. BurstGPT found working-hour patterns in its conversation-service traces and more irregular bursts in API traffic. Separately, the KV-cache testing suite preserves multi-request sessions, client delays, and nested subagents. A session trace is not necessarily an accepted task, and matching text hashes indicate potential reuse rather than actual cache residency. Use representative sampling to establish which workload these records describe.

Workload economics

Price accepted outcomes

Unit economics measures cost and value per meaningful unit of delivered work. Start with the attributable cost ledger, then state the task cohort, follow-up horizon, included costs, and allocation rules. Include model calls, tools, retrieval, storage, unsuccessful attempts, cancellation-related work, and material human review. Do not add provider charges twice through both request records and reconciled billing totals.

c_{\mathrm{attempt}}=\frac{C}{A},\qquad c_{\mathrm{completed}}=\frac{C}{D},\qquad c_{\mathrm{accepted}}=\frac{C}{K}.

Here

C

is the entire scoped cohort cost,

A

is executed attempts,

D

is terminally completed tasks, and

K

is known acceptable outcomes. Terminal completion includes finalized failures, not just success. Cancelled and unfinished tasks remain separately recorded, with their incurred costs included. Each ratio requires a nonzero denominator; no accepted outcomes makes cost per accepted outcome undefined, not zero.

A cheaper call, a dearer result

Consider this constructed comparison of the same 100 tasks. Each attempt makes one model call. Both configurations finish all tasks within the required time, all outcomes are assessed, and 90 are accepted. Review costs $1 per reviewed task; the remaining included cost is $10 per configuration.
Quantity	Configuration A	Configuration B
Attempts	120	150
Charge per model call	$0.10	$0.06
Model charges	$12	$9
Reviewed tasks / review cost	8 / $8	20 / $20
Other included cost	$10	$10
Total scoped cost	$30	$39
Cost per attempt	$0.25	$0.26
Cost per completed task	$0.30	$0.39
Cost per accepted outcome	$0.333	$0.433

B saves $3 in model charges but adds $12 in review, increasing total cost by $9. Its cheaper calls do not compensate for the changed workflow. The review amount is an assigned labor cost; whether reducing those hours would reduce payroll is a separate decision developed next.

Keep assessment coverage beside every outcome ratio. Recent tasks may not yet have downstream results, and selectively reviewed tasks may differ from unreviewed ones. Absence of a recorded failure is not acceptance. Compare cohorts at the same follow-up horizon and report unresolved counts. A definitive deadline miss can already be known even while eventual correctness remains unresolved. Incomplete-feedback analysis explains the limits of inference from such records.

Separate savings from allocation

Marginal cost is the added cost of more work. Fully allocated cost assigns a workload its share of all included expenses. Avoidable cost is future expenditure that changes because of a decision. These quantities differ: freeing already-paid staff time or committed compute creates capacity, but does not necessarily reduce cash expenditure. Scarce released capacity can still have value because it can perform other work.

The same reduction in processing work has different consequences under different purchasing arrangements.
Arrangement	Immediate consequence	When expenditure falls
Metered consumption	Fewer chargeable units	As avoided usage reduces charges under the applicable tariff
Existing capacity commitment	More spare capacity	When capacity can be reduced or a future purchase avoided
Removable serving units	Less work per unit	When enough capacity becomes removable without violating service requirements

Define the planning horizon before calling a cost fixed or variable. A server may be fixed for the current contract but removable at renewal; adding another required replica creates a step in cost. Allocate shared resources transparently, using measured consumption or staff activity where appropriate, and retain estimated or unallocated amounts. Ownership accounting can include acquisition, energy, support, software, and labor. Depreciation spreads an asset's cost over its useful life; do not also charge its full purchase price to the same period.

Use dated price rules and reconcile usage with billing, while preserving their coverage differences. Anthropic's documented Usage and Cost API, for example, has separate reports with different inclusions and reporting delays. Provider totals cannot supply the application's task-to-attempt mapping or external review costs. Include startup, idle capacity, telemetry, evaluation, and maintenance when they materially change the decision.

Time and consequential delay

Budget the completion path

A latency budget allocates an end-to-end allowance across necessary execution and waiting. Reuse the observation boundaries already established for the service. Time to first token, or TTFT, concerns the first token or content arrival at a named observation point; it need not be the first useful result. Streaming can improve early responsiveness without shortening final completion.

The critical path is the longest required dependency path determining earliest completion under the represented constraints. An operation starts no earlier than the latest finish among its prerequisites. Shared-resource serialization and additional waiting must also be represented; do not count prerequisite waiting again inside execution duration. Slack is the delay an activity can absorb without delaying completion under that schedule.

For example, suppose a request waits 100 ms for admission, then spends 600 ms generating a candidate. A 200 ms format check and a 400 ms evidence check start together once generation finishes. Release requires both checks and takes another 100 ms. Completion takes 100 + 600 + 400 + 100 = 1,200 ms: the checks overlap, so only the longer check determines when release can start. A 1,500 ms allowance leaves 300 ms of reserve without omitting either required check.

Parallel checks, one release boundary

The example finishes release at 1,200 ms, leaving 300 ms before the 1,500 ms allowance. Both checks require generation; release waits for both checks. The reserve is not executed work, and the schedule does not establish when a first useful output is exposed.

The placement of verification is part of the service contract. Hippocratic AI's June 2026 architecture account describes parallel supervisors: some block current outputs, including medication-guidance checks, while others can intervene on later turns. Moving a check off the current response path changes when protection applies. It does not make verification free or establish equivalent protection. Likewise, when human approval is required before completion, its queue and review time belong inside the budget.

Later operations inherit the remaining deadline, not a fresh allowance. Before another attempt, account for backoff, execution, and the work still needed afterward. Propagating the original deadline helps avoid consuming resources for a caller that has already stopped waiting.

Find the delay worth removing

First estimate how much of the completion path an intervention can change. Amdahl's central warning remains useful: acceleration of one portion leaves the rest. For a fixed workload, the familiar speedup bound makes that limitation explicit.

S=\frac{1}{(1-f)+f/s}.

Here

f

is the fraction of baseline duration accelerated,

s

is that portion's speedup, and

S

is whole-workload speedup. If 40% of a 10-second path becomes twice as fast, duration becomes

6+4/2=8

seconds: a 1.25× speedup. Even eliminating that portion leaves six seconds. This assumes unchanged remaining work, dependencies, overhead, and contention.

Dependency structure can impose a tighter limit. Accelerating one of two equally long critical paths leaves the other intact. Conversely, apparently noncritical work may matter if finishing it releases a scarce resource. After an improvement, bottleneck migration means a different constraint now determines completion; remeasure rather than applying the original fraction repeatedly.

Tail latency describes slow responses; p95, the 95th percentile, is a threshold at or above roughly 95% of observations. Better averages can hide deadline failures. Dean and Barroso's 2013 The Tail at Scale explains fan-out: parallel requests expose completion to slow dependencies. Which dependencies must finish, and whether their delays coincide, determines the effect.

Each example has 100 two-stage tasks, timed in milliseconds. A contains 94 pairs (1,1), three (10,1), and three (1,10); B contains 97 pairs (1,1) and three (10,10). Stage distributions match, each with p95 of 1 ms. Total p95 differs: A is 11 ms; B is 2 ms. Component percentiles cannot be added.

Same stages, different pairing

Example datasets with 100 tasks each: A has 94 totals of 2 ms and 6 of 11 ms; B has 97 totals of 2 ms and 3 of 20 ms. Stage distributions match. At the first observed total covering at least 95% of tasks, A has p95 = 11 ms and B p95 = 2 ms; B still has the larger maximum.

Prioritize an intervention by the delay it can remove, the tasks exposed to that delay, and its attainable improvement. Then check the complete latency distribution again. Reducing a frequent short operation and removing a rare deadline-breaking wait solve different problems.

Avoided work and valid reuse

Remove unnecessary execution

Additional computation can be valuable: another attempt may solve a difficult task, and verification may prevent a bad result. The target is work that does not justify its resources through better accepted outcomes. Distinguish removing an unnecessary model call, retaining a completed computation, sharing one already underway, and allowing another independent attempt.

Memoization retains results for later matching calls. Request coalescing shares an execution still in flight. Python's lru_cache can execute overlapping misses twice despite thread-safe bookkeeping. Go's singleflight runs one same-key call within a Group; duplicate callers wait for its result. The application must establish safe sharing. Neither persistent caching nor cross-process coordination follows from this contract.

Sharing a result requires equivalent result obligations. It does not authorize replaying an external action: safe repetition needs its own contract. Retries also add demand when a dependency may already be overloaded. Independent loops can multiply attempts across layers. Use retryable-error distinctions, backoff, per-request limits, and shared retry budgets; an attempt budget is not a monetary budget when operations have different prices.

Stopping the caller's wait is not necessarily stopping execution. The gRPC deadline contract makes the server application responsible for terminating activities it spawned after cancellation. Measure abandoned work rather than assuming its computation or billing vanished. Harness Engineering owns those controls; Reasoning and Test-Time Compute owns decisions to spend additional computation deliberately.

Changing where models are used can matter more than changing models. Fink and colleagues' May 2026 radiology report-labelling study retained rule-valid fields and sent only rule-invalid fields to a language model. Across paired comparisons of 22 models on 2,923 structured computed tomography (CT) pulmonary angiography reports from one regional reporting network, median cohort processing time fell from 6.7 hours to about one hour, and reported marginal cost from €1.19 to €0.17. The reference standard retained rule-valid fields, potentially favoring hybrids; costs excluded development and full operating overhead. This supports investigating call elimination, not assuming rules always win.

Define valid cache reuse

A cache entry retains an artifact for possible reuse. A cache hit finds an entry that satisfies both a matching rule and a validity rule. Matching proposes that the work corresponds; validity establishes that reusing it is still permitted and suitable. First identify the artifact, because different caches avoid different operations.


Reuse type	Retained artifact	Work avoided
Deterministic preprocessing	Transformation result	Repeating the same valid transformation
Exact response reuse	Completed answer	Generating another answer for an equivalent request
Semantic response reuse	Answer plus matching representations	Generation, if approximate matching preserves answer validity
Compatible prefix reuse	Numerical model state	Processing the already-computed prefix, not new generation

A key-value cache, or KV cache, stores numerical attention state derived from processed tokens. Cross-request prefix reuse requires compatible preceding state, not similar meaning. vLLM's versioned prefix-cache design includes preceding-block identity, exact token IDs, and additional identifiers such as adapters and multimodal content. Reusing this state skips some prefill; the model must still process new input and generate the requested continuation.

Embeddings are learned vector representations that can propose semantically similar requests. Similarity does not establish identical answers: the same question can refer to different records or conversation contexts. MeanCache compares conversation context after matching query representations, illustrating why the latest sentence alone is insufficient. Neither test establishes authorization or freshness. Representation-learning limits explain why nearby vectors can preserve the wrong distinctions.

An application-level validity contract must account for result-affecting inputs, transformation and model versions, configuration, source revisions, and required variability. Tenant and permission context must come from verified identity, not a model-proposed identifier. Recheck changed authority before disclosing retained results. A request requiring a fresh random sample or a new external effect cannot generally be satisfied by replaying an old return value.

Invalidation makes an entry ineligible after a relevant change. Eviction removes retained state to manage capacity. A time to live, or TTL, limits retention or reuse age under a specified policy; it does not promise that the source stayed unchanged. In the reported credit-decision incident, a database write succeeded but failed invalidation left a downstream agent using the old score. Faster retrieval of that value was not a successful optimization.

Measure the value of reuse

Temporal locality means information is reused near its previous use. A working set is the information referenced over a relevant interval. Denning's original model used recent process execution: retaining too little caused repeated retrieval, while retaining too much consumed memory. Replacement policy alone could not repair excessive aggregate demand. The modern application is to balance avoided recomputation against the space and time occupied by retained state.

A miss penalty is the work or delay incurred when reuse is unavailable. Request hit rate counts each request equally, regardless of the expense avoided. Facebook's 2013 Scaling Memcache distinguished access frequency from miss expense when allocating cache pools. It also addressed a cache stampede: many readers simultaneously fall back to expensive storage after invalidation. A lease allowed one client to refill a missing entry while competing readers briefly waited. If another invalidation occurred before that refill finished, the stale refill was rejected.

Measure avoided execution expense against lookup, fill, retention, refresh, transfer, and validation expense over the same interval. Track saved input processing, generated output, transferred bytes, charges, and completion time separately. A hit can save expensive computation without shortening a different critical path. Semantic reuse that returns the wrong answer remains a quality failure, even when it avoids every model call.

A bounded break-even calculation

For a fixed eligible prefix with ordinary input charge

B

, one cache write at multiplier

w

followed by

r

reads at multiplier

a

costs

B(w+ar)

. Without caching it costs

B(1+r)

. Prefix-charge savings require

r>\frac{w-1}{1-a},\qquad a<1.

In Claude's pricing inspected August 30, 2026,

a=0.1

, while five-minute and one-hour writes used

w=1.25

and

w=2

. These require respectively one and two subsequent reads. This excludes suffixes, outputs, tools, and cache-management overhead; expiry or changed prefixes can require another write.

Longer retention helps only when it spans enough valid reuse. In Context Platform Engineering, longer lifetimes bridge more pauses but increase the retained working set. GPU high-bandwidth memory (HBM), host memory (DRAM), and external storage offer different places to retain compatible KV state. A lower storage tier can hold reusable state yet deliver it too slowly to outperform recomputation. Test cold misses, warm reuse, transfer overhead, and mixed-tier concurrency separately; neither a larger cache nor a warm single-request result establishes a serving improvement.

Separate weights and prefix KV in GPU HBM. Alternative DRAM or storage transfers and token recomputation supply compatible KV before remaining input and new generation. — Compatible prefix state can already be resident, arrive from another tier, or be recomputed from prefix tokens. Saved prefill is useful only after accounting for management and transfer; remaining-input processing and generation still run. This is a conceptual architecture, not a universal serving topology.

Usable capacity

Distinguish activity from throughput

Utilization needs a named resource, definition, and interval. For a graphics processing unit (GPU), the NVIDIA Management Library (NVML) reports the percentage of a sampling interval with at least one kernel—a function executing on the device—running. This measures activity, not the fraction of peak arithmetic throughput achieved. Its memory-utilization field likewise measures memory activity over time, not allocated capacity. Neither metric establishes how much useful work completed: a busy device can be executing retries or producing late results.

Throughput is completed units per time. Goodput adds a compliance rule that must be stated: a benchmark may count requests meeting configured latency objectives, while an application may require accepted tasks delivered before a deadline. Those are not interchangeable. Report quality-assessment coverage separately when only some completions have known outcomes.

Concurrency can spread shared execution costs across requests, but every active request also needs state and competes for resources. The inference capacity envelope connects request sizes, memory, batching, and queueing. Bursts temporarily create more work than the service can finish, while unusually slow requests occupy capacity longer. Spare capacity lets the service catch up. Near saturation, little remains for recovery, so greater arrival or service-time variability can increase waiting even at unchanged average demand. There is no universal utilization percentage that guarantees acceptable latency.

John Little's 1961 queueing result connects consistent long-run averages:

L=\lambda W.

Here

L

is the average number of items inside the chosen boundary,

\lambda

is their arrival rate, and

W

is average residence time. If residence includes waiting and execution, occupancy must include both. At 10 requests per second and two seconds of average residence, average occupancy is 20 requests. This accounting relationship is not a prediction of tail latency or a remedy for an unstable overloaded queue.

Little's 2011 retrospective also treats finite observation windows. If work remains at either boundary, ordinary completed-request counts and full completion latencies cannot simply be substituted into the identity. The boundary population and time spent inside the window require explicit accounting.

The useful unit depends on the application. For streaming speech, generation must sustain playback. Once that requirement is met, faster token production may matter less than startup delay and the number of simultaneous playable streams. Optimizing Inference for Voice Models uses this distinction to explain why maximizing raw token speed can miss the economic objective.

Provision for bursts and recovery

Arrival rate describes incoming work per time. Ready capacity describes work the available service can actually process. Their mismatch accumulates a backlog. Headroom is spare capacity beyond current demand, while scaling lag is the delay before added resources become usable. Translate each task class into its model calls, input and output work, tool demand, and review demand; a single requests-per-second limit cannot represent unequal tasks.

Keep offered, admitted, completed, rejected, cancelled, and unfinished work separate. Rejection protects capacity but does not erase demand. Hosted quotas add further ceilings. Claude's rate-limit documentation, inspected September 1, 2026, separates request, input-token, and output-token limits and warns about bursts and acceleration limits. For most documented models, cache reads avoid input-token quota consumption, with exceptions; losing reuse can therefore raise both charges and quota demand. A permitted rate is not a throughput or latency guarantee.

Ready capacity must exceed new demand

Consider a simple model that treats work as continuously divisible and processes it in arrival order: first in, first out (FIFO). Two work units arrive per second, with no service until second three. Six units accumulate. Service at four units per second clears the backlog by second six; service at two preserves it. Backlog $B$ drains in $B/(C-r)$ because only capacity beyond new arrivals clears waiting work. This requires constant service $C>r$ and continuing arrivals $r$ ; average benchmark throughput alone does not guarantee that service rate.

Readiness and headroom determine recovery

A lossless FIFO fluid model: arrivals stay at 2 work units/s, with no service before readiness. Work units are continuously divisible; this is not a forecast of AI request percentiles.

Ready service (work units/s)Startup lag (s)

At readiness: 6 queued units. Drain duration after readiness: 3 s; absolute catch-up: 6 s.
FIFO delay at cumulative work unit 2: 2.5 s. Backlog at 10 s: 0 units.

Time (s)	Arrived	Completed	Backlog
0	0	0	0
3	6	0	6
6	12	12	0
10	20	20	0

With positive backlog and C > 2, drain duration is B/(C−2). Once caught up, completed work follows arrivals even though ready capacity can be higher. Default: lag 3 s and service 4 units/s create 6 queued units, drain them over 3 s, and catch up at 6 s; work unit 2 waits 2.5 s.

Arrivals remain fixed at 2 work units/s. Vertical separation is unfinished work; horizontal separation at equal cumulative work is FIFO delay. Available service must exceed arrivals to drain positive backlog. The model does not infer request percentiles or warm-capacity prices.

Good warm execution cannot compensate for arbitrary startup delay. One RunPod deployment demonstration reported about 41 seconds queued and 1.5 seconds executing, attributing part of the wait to initialization. That is one request, not a latency distribution. Test actual scaling events using the startup boundary, including recovery after capacity loss.

Warm capacity exchanges idle expense for lower startup exposure. Modal's scaling controls distinguish minimum warm containers, an active-service buffer, and idle scale-down duration. Their value depends on demand gaps and startup behavior. Deferrable work can wait for capacity; interactive work may need ready headroom before the burst arrives.

Pay for the required service

Compare purchasing arrangements only after establishing the required operating envelope: workload mix, quality, latency, availability, and recovery behavior. Total cost of ownership includes acquiring and operating the resources, not just their headline rate. Use measured effective capacity under those requirements, carrying forward necessary headroom, redundancy, startup, labor, maintenance, commitments, and billing granularity.

Elasticity changes the amount purchased as well as its price. The Berkeley cloud report illustrated demand peaking at 500 servers but averaging 300: peak provisioning buys 12,000 server-hours daily for 7,200 server-hours of demand. Its hypothetical example explains how a higher metered hourly price can still produce lower expenditure. Modern AI comparisons must additionally include readiness and workload-specific capacity rather than assuming instantaneous scaling.

Commitments change what can be avoided. Bedrock Provisioned Throughput documents hourly-billed model capacity and commitments that cannot be deleted before their term ends. Lower usage can leave the bill unchanged. Conversely, an asynchronous service can sell scheduling flexibility: Claude's batch-processing documentation, inspected August 30, 2026, describes half-standard API pricing and expiry of unfinished batches after 24 hours. This is neither faster interactive completion nor guaranteed success for every request.

**Sensitivity analysis** varies uncertain assumptions to determine when a decision changes. For each arrangement, recompute cost only after checking feasibility in these scenarios.
Demand scenario	Feasibility check	Economic sensitivity
Sparse interactive traffic	Cold-start exposure versus deadline	Minimum warm capacity and operating labor
Stable sustained traffic	Measured service capacity with required headroom	Commitment terms and removable capacity units
Bursty interactive traffic	Readiness lag and backlog recovery	Paid reserve versus metered burst capacity
Deferrable background work	Completion window and failure handling	Scheduling discount versus delay

There is no universal demand threshold at which self-management wins. A comparison can change when operating labor increases, a replica becomes removable, or required headroom rises. Local and On-Device AI adds device-specific constraints; here those constraints enter as feasibility and cost inputs.

Optimization experiments

Test the complete workload

Write the expected mechanism before the comparison: what work disappears, what wait shortens, or what capacity becomes usable. Specify the baseline, cost scope, minimum worthwhile improvement, acceptable quality loss, timing targets, and rejection conditions. A paired comparison evaluates baseline and candidate on the same sampled tasks. Preserve the task pairing and relevant groups when analyzing differences; matched-work evaluation develops the method. Similar average scores alone do not establish that a quality-loss limit was met.

An optimization result should retain enough context to distinguish a mechanism gain from a changed workload.
Record	What must remain interpretable
Workload and system	Task mix, allowed attempts, versions, resource guarantees and limits
Reuse conditions	Cold versus warm state, prefix overlap, working-set size, and replay order
Load coverage	Configured arrivals, started work, generator omissions, and server rejections
Results	Accepted outcomes, assessment coverage, cost, timing distribution, failures, cancellations, and unfinished work

Isolated tests answer narrower causal questions. Callan Fox's October 2025 cache-testing method first compares cold and cached requests, then tests external-manager overhead while data still fits GPU memory, then external retrieval, and finally mixed tiers under concurrency. That sequence separates reuse benefit from background-transfer cost. It must still be followed by the whole workload under representative load.

Load generation can hide overload. In a closed model, a user waits for its previous operation before starting another, so slower service reduces offered traffic. An open model schedules independent arrivals. Either can represent a real workload, but they answer different questions; see open-loop and closed-loop load. An arrival-rate test also needs enough generator capacity: k6 records dropped iterations when it cannot start scheduled work. Those are omissions by the generator, not automatically server rejections.

Repeatability can narrow realism. MLPerf's end-to-end RAG benchmark measures tasks per second across a multi-step workflow. Retrieval-augmented generation, or RAG, retrieves evidence for generating an answer. The benchmark's performance runs fix intermediate inputs and hop counts while still generating outputs, with accuracy evaluated separately. This controls execution variation but does not measure unconstrained live trajectories; its initial offline workload also does not establish interactive tail latency.

Finally, keep environment effects visible. Anthropic's coding-evaluation infrastructure study held model, harness, and tasks fixed while varying resources. Extra headroom could prevent infrastructure failures; larger allocations could also enable different strategies. Record both guarantees and hard limits, repeat comparisons, and retain task-class differences. A result produced under more permissive conditions is not automatically an improvement to the same system.

Compare interventions across layers

Routing selects a model or provider. A cascade calls stages sequentially, using an earlier result to decide whether to escalate. The available decision information differs: RouteLLM chooses from the query before generation, whereas FrugalGPT scores generated answers before deciding to continue. Strong-model call share is a cost proxy, not total expenditure. Model Routing and LLM Gateways owns the selection and fallback policies.

For a two-stage cascade, let

c_1

be first-stage cost,

c_v

the cost of scoring its answer,

c_2

second-stage cost, and

e

the escalated fraction. With fixed per-stage costs:

\mathbb{E}[C_{\mathrm{calls}}]=c_1+c_v+e c_2.

Initial-only tasks pay

c_1+c_v

; escalated tasks pay all three terms. Escalation does not refund the first attempt. Variable request sizes require per-task accounting. Review, failures, and deadline outcomes remain separate parts of the whole-workload comparison.

Other interventions change different quantities. Context Engineering selects and reduces model inputs. Quantization changes numerical representation and execution requirements. Distillation trains a student using a teacher's behavior, potentially replacing later teacher calls. These are not interchangeable discounts: each changes its own computation, quality risks, and development cost. Test combinations end to end because their benefits and overheads can interact.

Optimization itself consumes resources. In AI Agents That Matter, joint prompt and configuration optimization reduced variable cost by 53% for the tested GPT-3.5 pipeline and 41% for Llama-3-70B versus default DSPy approaches, with similar measured retrieval performance. The held-out experiment used 200 cases across five runs. Retrieving the required documents is not final-answer correctness, and these historical costs omit a complete production workflow. The enduring distinction is upfront optimization expense versus recurring run cost.

Estimate payback over a plausible service life, including maintenance. In a simple declared scenario, a $2,000 implementation plus $500 of additional maintenance requires 50,000 tasks to break even; more tasks produce net savings if avoidable savings remain $0.05 per task. That calculation is useful only while quality, workload mix, savings, and demand assumptions hold. Released capacity under an unchanged commitment is not the same as those cash savings.

Adoption and maintenance

Choose and revisit the operating point

First reject candidates that violate the workload contract. Among feasible choices, one dominates another if it is no worse on every stated objective and better on at least one. The Pareto frontier contains evaluated choices with no dominator. It need not identify a single winner.

The candidate plot uses declared example values at one fixed workload. All conditions other than the labeled quality and deadline failures are assumed satisfied. E has some accepted outcomes, so its cost per accepted task is defined, but it fails the required quality threshold. The p95 target of at most 1.5 seconds concerns the population distribution, not every individual task. A and B trade cost against latency; C is worse than an available alternative. D and E are ineligible before preference enters the decision. Real estimates also have uncertainty: overlapping or unstable measurements may not establish dominance.

Feasibility precedes preference

Among these evaluated feasible choices, A and B each dominate C but neither dominates the other. D misses the population p95 target and E fails quality. Discrete points do not establish an unmeasured frontier between configurations.

Use a canary—a bounded, time-limited live deployment compared with a control—to test remaining operational uncertainty before wider adoption. Define exposure, acceptance and stop conditions, and rollback in advance. A small canary can miss rare failures or absent task classes. Choosing a live experiment distinguishes this from shadow execution, which does not apply the candidate's results to the real workflow.

A reusable decision procedure follows from the chapter's measurements.

Confirm the contract — Identify the workload, accepted outcome, timing boundaries, review limit, cost scope, and unresolved measurements.
Identify the mechanism — Name the computation, delay, retained state, or purchased capacity that the intervention changes.
Reject infeasible changes — Keep quality, authorization, deadlines, capacity, and review constraints ahead of cost preference.
Compare the remaining choices — Use whole-workload results, uncertainty, operating effort, and payback sensitivity rather than nominal prices.
Validate bounded exposure — Check live outcomes against the control and stop when prespecified limits are breached.
Record revisit triggers — Reassess when task mix, demand, model behavior, prices, commitments, cache reuse, or review burden changes.

Unit cost and total spending answer different questions. In a simple example, halving cost per accepted task while tripling accepted volume raises total cost by 50%. That can be successful efficiency with growing demand, not a regression. The durable achievement is an understood operating point: what it delivers, what it costs, which conditions make it feasible, and when those conditions must be checked again.

Open questions

Adaptive stopping must estimate the value of another attempt while task difficulty and failure causes change. A promising stopping policy can save computation yet abandon cases that remain recoverable. Progress would mean held-out, shifted-workload evidence connecting incremental attempts to accepted outcomes, review burden, and deadline compliance—not confidence statements alone.
Cache placement must balance reuse against congestion. The worker with the most compatible state may also have the longest queue, while moving that state has its own cost. Progress would mean policies evaluated under changing task mixes that improve completion latency and cost together without relying on privileged knowledge of future requests.
Forecasting capacity for autonomous workflows remains difficult because the workflow creates part of its own demand: retries, delegation, and growing histories change future calls. Useful progress would combine task-level trace structure with validated forecasts of bursts and recovery, retaining uncertainty instead of treating an average session as a fixed request multiplier.
Whole-workflow economics depends on delayed and selectively observed outcomes. Cheap automatic completions may receive less scrutiny than expensive escalations, making their apparent efficiency difficult to compare. Progress would include comparable follow-up and assessment coverage, with unresolved outcomes visible and conclusions tested against plausible missing-outcome assumptions.

Cited in this entry

Explores delegation and context sharing in sessions whose difficulty changes, motivating whole-session accounting rather than a permanent cheap-versus-expensive model classification.

Watch talk

Explore more talks

Search the talk library

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

30 matching talks

Talk	Speaker	Event	Year
Frontier results, on device - RL Nabors, Arize Transcript reviewed	Rachel Lee Nabors (RL Nabors)	AI Engineer World's Fair 2026	2026
Engineering voice agents: Latency, quality, and scale Transcript reviewed	Rishabh Bhargava	AI Engineer Europe 2026	2026
Lessons from building GenAI based applications — Juan Peredo Cited in this entry	Juan Peredo	AI Engineer Summit 2025	2025
200 Million Patient Interactions Later: What the Generic Voice Stack Misses Transcript reviewed	Vivek Muppalla	AI Engineer World's Fair 2026	2026
RAG and the MongoDB Document Model Cited in this entry	Ben Flast	AI Engineer World's Fair 2024	2024
Voice In, Visuals Out: The Agony and the Ecstasy Transcript reviewed	Allen Pike	AI Engineer World's Fair 2026	2026
From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik Cited in this entry	Sandipan Bhaumik	AI Engineer Europe 2026	2026
Building Context-Aware Reasoning Applications with LangChain and LangSmith Transcript reviewed	Harrison Chase	AI Engineer Summit 2023	2023
Scaling the Next Paradigm of Heterogeneous Intelligence Transcript reviewed	Adrian Bertagnoli	AI Engineer Europe 2026	2026
Building Deterministic Infrastructure for Non-Deterministic AI Agents Cited in this entry	Nishant Gupta	AI Engineer World's Fair 2026	2026
Infra behind Krea 2 - How to train and serve at scale Transcript reviewed	Gabriel Jorge Menezes	AI Engineer World's Fair 2026	2026
Efficient Reinforcement Learning Transcript reviewed	Rhythm Garg, Linden Li	AI Engineer Code 2025	2025
Computer-Use 2.0: Agents Just Got Multi-Cursor Transcript reviewed	Francesco Bonacci, Dillon DuPont, Robert Wendt	AI Engineer World's Fair 2026	2026
Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI Transcript reviewed	Keegan McCallum	AI Engineer World's Fair 2025	2025
The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked Transcript reviewed	Filip Makraduli	AI Engineer World's Fair 2025	2025
Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod Cited in this entry	Audry Hsu	AI Engineer Europe 2026	2026
How Zapier Builds AI Products and Features With the Help of Braintrust Transcript reviewed	Ankur Goyal, Olmo Maldonado	AI Engineer World's Fair 2024	2024
Using RL-based Agent to Detect and Remediate ETL Pipeline Failures Cited in this entry	Anna Marie Benzon	AI Engineer World's Fair 2026	2026
How Web Data Infrastructure Powers the Next Generation of AI Cited in this entry	Patricija Žemaitytė	AI Engineer World's Fair 2026	2026
Playground in Prod - Optimising Agents in Production Environments Transcript reviewed	Samuel Colvin	AI Engineer Europe 2026	2026
Vector Search Benchmark[eting] Transcript reviewed	Philipp Krenn	AI Engineer World's Fair 2025	2025
How to Kill the Code Review Transcript reviewed	Ankit Jain	AI Engineer World's Fair 2026	2026
Fast Models Need Slow Developers Transcript reviewed	Sarah Chieng	AI Engineer Europe 2026	2026
Your Agent Didn’t Fail. Your Harness Did. Transcript reviewed	Vinoth Govindarajan	AI Engineer World's Fair 2026	2026
AI Engineering 201: Inference Cited in this entry	Charles Frye	AI Engineer Summit 2023	2023
Keynote: The AI developer experience doesn't have to suck – why and how we built Modal Transcript reviewed		AI Engineer Summit 2025	2025
What Lies Beneath the API — Benjamin Cowen, Modal Transcript reviewed	Benjamin Cowen	AI Engineer Europe 2026	2026
20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna AI Transcript reviewed	Bertrand Charpentier	AI Engineer Europe 2026	2026
Backlog.md: Terminal Kanban Board for Managing Tasks with AI Agents — Alex Gavrilescu, Funstage Cited in this entry	Alex Gavrilescu	AI Engineer Code 2025	2025
Building Turbopuffer: Gergely Orosz (@pragmaticengineer ) × Simon Eskildsen (CEO) Transcript reviewed	Gergely Orosz, Simon Eskildsen	AI Engineer World's Fair 2026	2026

References

Coverage and source review

Processed transcripts: 35 processed in full · 5 in the curated path
Automated source review: Passed
Metadata candidates: 0 unreviewed; not verified topic membership
Corpus version: 1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

NIST/SEMATECH e-Handbook: Percentiles
A percentile locates a threshold within ordered observations; interpolation conventions can differ. Constructed application of NIST's percentile calculation: consider 100 two-stage tasks, with durations in milliseconds. If 94 have stages (1,1), three have (10,1), and three have (1,10), each stage's p95 is 1 ms, but total-duration p95 is 11 ms. With 97 tasks at (1,1) and three at (10,10), the stage distributions are unchanged, yet total p95 is 2 ms. Thus component percentiles do not determine the end-to-end percentile without their joint behavior.
Le Boudec and Thiran: Network Calculus
An arrival curve bounds work entering during an interval; a service curve describes guaranteed service. Their separation bounds backlog and delay. For arrivals bounded by burst b plus sustained rate r, and service guaranteed at rate R after latency T, the book gives backlog bound b+rT and delay bound T+b/R when R is at least r. Larger service latency therefore requires more buffering even without higher sustained demand. Derived constant-rate application: an existing backlog B drains in B/(C-r) time when service remains C>r and arrivals continue at r.
Boyd and Vandenberghe: Convex Optimization Slides
Multicriterion optimization separates feasibility constraints from competing objectives. A feasible point dominates another when it is no worse on every objective and strictly better on at least one. A Pareto-optimal point has no dominating feasible alternative; multiple such points can form a tradeoff curve or surface. Scalarization combines objectives with explicit relative weights.
Backlog.md: Terminal Kanban Board for Managing Tasks with AI Agents — Alex Gavrilescu, Funstage
Acceptance criteria should describe testable behavior, and completion should depend on satisfying the definition of done.
Hacking the Inference Pareto Frontier
Set required quality and latency from the application experience, then minimize cost within those constraints.
Google SRE: Service Level Objectives
A service-level indicator is a defined quantitative measurement of service behavior; a service-level objective sets its target value or range. Measurements are aggregated over a specified window. Client-observed latency can differ from server-observed latency. Increasing demand can increase latency and eventually expose a performance cliff. Targets therefore need measurement boundaries and operating conditions, not just a number.
Google SRE Workbook: Implementing SLOs
SLO design starts with identified users, their critical activities, and a diagram of request flows, data flows, and dependencies. The chapter distinguishes request-response services from pipelines that transform records over potentially much longer intervals. It recommends multiple latency thresholds to capture typical and slow experiences: a target covering most requests can still leave a substantial minority waiting much longer.
Kelley and Walker: Critical-Path Planning and Scheduling
The paper represents required activities as an acyclic network with finish-to-start dependencies. For stated durations, the earliest event time is the maximum of predecessor event time plus connecting activity duration. This recurrence yields the longest start-to-finish path and the earliest possible completion when activities can start as soon as their prerequisites finish. Delivery restrictions are represented as activities rather than omitted waiting. Derived implication: shortening an operation leaves completion unchanged if an unchanged longest path remains, provided other durations and dependencies stay fixed. With several equally long paths, accelerating only one need not shorten completion.
Amdahl: Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities
At the 1967 Spring Joint Computer Conference, IBM's Gene Amdahl challenged claims that substantial computing advances required multiple cooperating processors. He argued that sequential data-management work and irregular application behavior constrained gains from accelerating parallel arithmetic. His comparison considered complete problems and hardware costs rather than arithmetic speed alone. This supplies the historical motivation for examining the unchanged portion of a workload before predicting an overall speedup.
Peter J. Denning: The Working Set Model for Program Behavior
Denning's MIT paper, presented in October 1967 and published in May 1968, connected processor scheduling with memory demand. It defined a working set as information referenced within a recent interval of process execution. Retaining too little causes repeated retrieval of still-needed information; retaining too much wastes memory. The paper argued that replacement policies alone cannot resolve excessive aggregate memory demand. Its retention-window analysis makes the tradeoff explicit: balance returning-page traffic against occupied memory rather than maximizing either retention or processor activity independently.
Armbrust et al.: Above the Clouds—A Berkeley View of Cloud Computing
Berkeley's February 10, 2009 technical report framed cloud elasticity as a way to reduce both idle provisioning and the risk of insufficient capacity. It contrasted resource acquisition in minutes with purchasing and installing equipment over weeks. Its constructed daily-demand example requires 500 servers at peak but 300 on average: fixed peak provisioning purchases 12,000 server-hours per day for 7,200 server-hours of demand. This illustrates why a higher metered hourly price can still yield lower workload expenditure. The report also calls for comparing operating costs and realistic average and peak utilization.
Kapoor et al.: AI Agents That Matter
The 2024 paper distinguishes one-time agent-optimization expenses from costs incurred on each subsequent run. Its HotPotQA experiment searches prompt demonstrations and configuration settings using separate development subsets, then evaluates 200 held-out cases across five runs. Joint optimization reports 53% lower variable cost for the GPT-3.5 pipeline and 41% for Llama-3-70B compared with the default DSPy approaches, with similar measured retrieval performance. The mechanism illustrates spending more during optimization to reduce recurring work, making deployment volume relevant to whether the effort pays back.
FinOps Foundation: How to Build a Generative AI Cost and Usage Tracker
Counting calls alone can misattribute token-priced costs when request sizes differ. The guide distinguishes estimated token counts from provider-reported input and output counts and recommends recording usage per API call before rolling it up by use case. Its illustrative records include request identity, model, API version, timestamp, and price-effective date. It warns that retaining raw prompts and outputs for cost estimation adds storage expense and sensitive-data exposure.
How fast are LLM inference engines anyway?
The demonstrated workload achieved higher request throughput when most tokens were input context rather than generated output.
Building Deterministic Infrastructure for Non-Deterministic AI Agents
Uncontrolled agent retries can amplify a small tool error into escalating compute consumption.
BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems
BurstGPT characterizes request concurrency, request and response lengths, and failures in real serving traces. Its conversation-service traffic shows working-hour and weekday patterns, while API traffic exhibits more irregular bursts. Burstiness also changes within the day. The authors argue that synthetic or non-LLM workloads can produce misleading serving evaluations because they omit these demand characteristics.
Inference Server Cache Performance Testing Suite
The repository describes coding-session traces with per-request server time, client delay and nested subagent structure. Its reported requests-per-trace distribution has quartiles of 19, 48 and 101, illustrating why one request is not an adequate model of a session. Cache reuse estimates come from hash overlap between successive requests within each conversation. A separate scenario assumes a partially warm shared prefix on the first request. These are different reuse assumptions, not interchangeable measurements of deployed cache hits.
Using RL-based Agent to Detect and Remediate ETL Pipeline Failures
Correct anomaly flags, fast successful recoveries, and low escalation are distinct properties and should not be conflated.
FinOps Foundation: Unit Economics
Unit economics connects technology spending with delivered value. FinOps distinguishes resource-efficiency metrics, such as cost per token, from business-unit metrics, such as cost per case resolved. It calls for documented unit definitions, data sources, calculation assumptions, cost inclusions, and allocation rules supporting fully loaded costs. Outcome goals, demand assumptions, and sustainable cost-to-serve targets should inform decisions; consistent trends within a defined scope are often more useful than comparisons across unrelated products.
Claude Platform: Usage and Cost API
Anthropic documents separate usage and cost reports for reconciling internal records with provider billing. Usage distinguishes uncached input, cache reads, cache creation, output tokens, and server-tool usage. Cost reports include token, web-search, and code-execution charges, with daily buckets. Coverage differs: code execution is absent from the usage endpoint, while Priority Tier costs are absent from the cost endpoint. Reports require pagination and can arrive after request completion.
National Academies: informative censoring and sensitivity analysis
An outcome can remain unobserved because follow-up ends before the event, or because the subject disappears from observation. Censoring can be informative when the reason observation ends is related to the event time. A scheduled cutoff and dropout need not have the same missingness mechanism. Time-to-event inference requires justified assumptions about event and censoring dependence, potentially conditional on recorded prognostic variables; sensitivity analysis explores remaining dependence. Applied to AI workflows, recent requests have had less time to receive downstream outcomes. Compare cohorts at a specified follow-up horizon, report unresolved counts, and avoid counting absence of a recorded failure as success.
National Academies: inference with missing outcomes
If O indicates an observed outcome and Y indicates success, the observed rate estimates P(Y=1|O=1), not automatically P(Y=1). Selective feedback can make these differ. Missing completely at random means missingness is independent of measured and missing values; missing at random permits dependence on observed information but not remaining missing outcomes after conditioning. MAR therefore does not justify an unadjusted complete-case success rate. Adjustment or weighting needs justified models and adequate observation probabilities. Dependence on unobserved outcomes requires additional assumptions and sensitivity analysis. Derived bookkeeping: with S successes, F failures and U unresolved eligible cases, the eventual success fraction lies between S/N and (S+U)/N, where N=S+F+U.
ACCA: Relevant costs
Relevant-cost analysis asks which future cash flows change because of a decision. Reallocating existing fixed costs does not itself change expenditure, while additional fixed costs can matter. ACCA's labor example distinguishes already-paid spare hours, additional temporary hiring, and scarce staff diverted from valuable work. Application to AI optimization: reducing review hours or freeing committed compute can create useful capacity without immediately reducing payroll or contracted charges. Fully allocated workload costs and incremental savings therefore answer different questions.
FinOps terminology: ownership, depreciation and utilization
FinOps defines ownership cost broadly, including acquisition, management/support, communications and labor; depreciation distributes asset cost over time, while activity-based costing can allocate staff hours times hourly rates. Illustrative engineering model: period cost=a(H-S)/L+energy+maintenance+staffing+software/network+service charges, where H is hardware purchase cost, S assumed residual value, L useful life in matching periods and a the workload's allocated share. Do not count both the full purchase and its depreciation in the same period model. With sustained busy throughput q, available time T and utilization u, completed volume is approximately q*u*T under a stable workload; fixed cost per unit therefore rises as utilization falls. Energy expense is measured kWh times the applicable tariff.
AI Engineering 201: Inference
Convert achieved token throughput and compute expenditure into dollars per token before comparing self-hosting with an inference service.
Amazon Bedrock: Increase model invocation capacity with Provisioned Throughput
Bedrock documents Provisioned Throughput as hourly-billed model capacity. Model Units specify model-dependent input-token processing and output-token generation capacity per minute. Price depends on the model, unit count, and commitment duration. The documented options include no commitment, one month, and six months; committed capacity cannot be deleted before its term ends, and billing continues until deletion. Application implication: reducing calls need not immediately reduce provisioned-capacity expenditure.
How Web Data Infrastructure Powers the Next Generation of AI
At high throughput, logs and metrics become part of the workload that the infrastructure must sustain.
FinOps Foundation: Usage Optimization
FinOps evaluates usage optimization against functional and nonfunctional requirements, not expenditure alone. It calls for comparing expected savings, cost avoidance, and efficiency gains with engineering effort, implementation risk, service disruption, and organizational priorities. Resources should match actual usage patterns and operate only when needed. This supports including the cost of making and maintaining an optimization when deciding whether its recurring savings justify adoption.
AIPerf Metrics Reference
AIPerf measures TTFT from request start to the first nonempty content response. Its metric named ITL is a per-request average: (request latency minus TTFT)/(output sequence length minus one), requiring at least two output tokens. It separately measures inter-chunk arrival gaps. The displayed request-latency formula ends at the last content response, so it does not independently establish a later terminal-event timestamp. Goodput counts requests satisfying configured SLOs per benchmark second; successful requests and errors have separate counters.
Metrics design — vLLM
Serving telemetry distinguishes queue time, prefill, decode, time to first token, inter-token latency and end-to-end latency. Event boundaries and observation location matter: client network time differs from engine intervals. A successful finish reason does not establish task correctness. Request lengths, waiting/running requests and KV usage help explain a latency change.
Project Management for Construction: Fundamental Scheduling Procedures
Basic critical-path scheduling assumes fixed activity durations and no resource constraints beyond represented precedence. Resource scarcity can make its minimum completion time unattainable. The text models resource serialization by adding a precedence relation between activities sharing equipment; required waiting between activities can instead be represented as a timed lag. Application to workflow timing: prerequisite waiting is already captured by taking the latest predecessor finish, while additional waiting needs an explicit constraint or interval. Avoid counting that same wait again inside an operation's duration. Shortening apparently noncritical work can matter when it releases a shared resource, so the precedence-only conclusion requires unchanged resource interactions.
Muppalla, Perkari and Jhaveri: Building AI for Healthcare Conversations, Part 2
In their June 18, 2026 architecture account, Hippocratic AI engineers describe a conversational agent supported by parallel specialist supervisors. Some supervisors synchronously block outputs, including checks on medication guidance; others monitor asynchronously and may intervene on later turns. This makes the latency decision concrete: which checks must finish before the current response is released, and which operate outside that response's critical path. The authors identify reconciling concurrent model judgments within the response allowance as an engineering problem.
Google SRE: Addressing Cascading Failures
Retries add demand precisely when an overloaded dependency has reduced capacity. Independent retry loops at several layers multiply attempts: the chapter's constructed example produces 64 database attempts from four attempts at each of three layers. Recommended controls include randomized exponential backoff, per-request limits, a server-wide retry budget, and distinguishing permanent from retryable failures. Propagating the original deadline and checking remaining time before later stages avoids spending resources on work whose caller has already stopped waiting.
CUDA C++ Best Practices Guide
Kernel launches return before GPU work completes; CPU timing must account for completion. CUDA events record device timestamps when reached in a stream, and the example waits for the stop event before reading elapsed time. Extra synchronization can alter execution. Bandwidth is data transferred per unit time. Occupancy is active warps per multiprocessor divided by its supported maximum; registers, shared memory, and block size constrain residency, and higher occupancy need not improve performance. Amdahl's law bounds fixed-workload improvement by the fraction accelerated: unchanged work remains even if the optimized portion becomes arbitrarily fast.
Dean and Barroso: The Tail at Scale
Shared-resource contention, maintenance, and queueing create occasional slow operations. When a request waits for many parallel dependencies, individual outliers can dominate its completion time. Hedging sends a duplicate request after a delay, accepts the first response, and cancels remaining requests. Delaying the duplicate limits extra load, but cancellation can race with execution, leaving duplicate work. The paper presents both hypothetical fanout calculations and measured distributed-storage examples.
Lessons from building GenAI based applications — Juan Peredo
A secondary classifier can screen requests or responses, but it adds latency and cost without guaranteeing correct classification.
Python functools: lru_cache
Memoization retains a function's computed result for reuse with matching arguments. Python's lru_cache bounds retained entries and exposes hits, misses, occupancy and explicit clearing. Its documentation excludes cases where callers require side effects, distinct mutable results, current time or fresh randomness. It retains references to arguments and results until eviction or clearing. Thread-safe cache bookkeeping does not suppress every duplicate computation: simultaneous callers can both execute before the first result has been cached. Thus completed-result caching and in-flight request coalescing provide different guarantees.
Go singleflight package documentation
singleflight suppresses simultaneous duplicate function calls within a Group. For a given key, Do allows one execution in flight; duplicate callers wait and receive its result. The published example starts two overlapping calls with the same key and shows both receiving the first function's result. This demonstrates request coalescing: sharing work already underway, distinct from retrieving a result retained from an earlier completed operation.
RFC 9111: HTTP Caching
A stored HTTP response is reusable only when method, target URI, nominated request headers, and freshness or validation requirements permit it. Shared reuse of authenticated responses has additional restrictions. Unsafe requests must be forwarded to the origin and receive a corresponding response before the cache replies. Eligible concurrent misses may be collapsed into one forwarded request; if the returned response cannot satisfy all waiting requests, forwarding them later can add latency.
gRPC: Deadlines
A deadline specifies when the client is no longer willing to wait; a timeout specifies an allowed duration. After expiry, the client can fail with DEADLINE_EXCEEDED and the server RPC can be cancelled, but the server application remains responsible for stopping activities it spawned. Deadline propagation deducts elapsed time before downstream calls. The published sequence illustrates a two-second allowance becoming 1.5 seconds after initial processing.
Fink et al.: How green are large language models for radiology report labelling?
Published May 27, 2026, this study compared workflows extracting the same 14 fields from 2,923 structured CT pulmonary angiography reports. Its hybrid retained rule-valid fields and sent only rule-invalid fields to an LLM. Paired comparisons across 22 models reduced median cohort processing time from 6.7 hours to approximately one hour and reported marginal cost from €1.19 to €0.17. Mean report-level accuracy was 98.5% for hybrids versus 85.1% for LLM-only pipelines. The intervention changed where models were used, not merely which model was selected.
MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Services
MeanCache first compares a new query with cached query embeddings, then compares the associated conversation context before reusing a response. A similar query without sufficiently similar context is forwarded to the model. Its design tunes similarity thresholds using cache classification performance and includes embedding computation, matching, and storage as overhead. This provides a concrete example of approximate answer reuse requiring more than similarity of the latest sentence.
Automatic prefix caching — vLLM
A repeated shared token prefix can reuse stored KV state and skip computation for that prefix. Stable initial context favors reuse; unrelated prefixes cannot share those computations merely because their meaning is similar. The benefit is in prefill, not generating new decode tokens, so long outputs may limit the fraction of request latency saved.
Automatic Prefix Caching — vLLM v0.14.1
vLLM identifies reusable full KV blocks using the parent block's hash, exact tokens in the current block and additional identifiers such as LoRA adapter IDs, multimodal input hashes and cache salts. Including preceding-block identity prevents an identical suffix from being mistaken for equivalent context. Image placeholders alone are insufficient: image content contributes a separate hash. An optional salt enters the first block's identity, restricting reuse to requests with the same salt.
OWASP Multi-Tenant Security: tenant context and resource access
Tenant context should originate in verified authentication claims or session state, not an unvalidated header or model-proposed tenant ID. Validate tenant status, propagate that context through application layers, and reset request-local context after use. Resource lookup combines tenant identity with resource identity and enforces authorization at the data-access layer. Row-level database policies and tenant-scoped repositories can prevent a valid caller from selecting another tenant's record merely by changing its ID. Cache keys and storage access need the same separation.
From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik
A shared cache can break agent-to-agent data consistency even when the underlying database write succeeds.
Claude prompt caching: matching and eligibility
Prompt caching reuses processing of a previously cached prompt prefix; it does not replay a stored answer. The prefix includes tools, system content and messages through the cache breakpoint. Hits require identical prefix content, including images, compatible request settings, an unexpired entry and the provider's cache isolation scope. Prefixes must meet the model-specific minimum token length; shorter requests are processed without caching. Entries become available when the first response begins. Explicit caching supports four breakpoints and searches up to 20 block positions backward for previously written entries. The default lifetime is five minutes, with a one-hour option; reuse refreshes lifetime. Usage fields distinguish cache creation and reads. Engineering implication: a hit proves prefix reuse, not that the source document or external world remains current.
Scaling Memcache at Facebook
The paper describes a thundering herd as many readers falling back to expensive storage after invalidation. Its lease mechanism permits a client to refill a missing key, rejects fills invalidated by intervening deletes, and makes competing readers wait briefly. Cache pools also distinguish access frequency from miss expense: infrequently requested keys can justify substantial cache space when misses are costly. Different churn patterns can otherwise evict still-valuable entries.
Callan Fox: Evaluating management of KV Cache within an inference system
Fox's October 31, 2025 methodology separates cache checks into distinct experiments. First compare cold and cached single requests. Then keep the working set within GPU cache capacity while enabling and disabling external cache management, isolating its background transfer overhead. To test retrieval from an external tier, make the working set exceed GPU capacity but fit the external tier. Finally test blended tiers under concurrency. The report emphasizes that retrieving cached state must be fast enough to beat recomputation; a single warm-request result cannot establish system-level benefit.
Claude Platform pricing: Prompt caching
The inspected pricing documentation charges five-minute cache writes at 1.25 times base input price, one-hour writes at twice base input price, and cache reads at 0.1 times base input price. For an unchanged eligible prefix whose ordinary input charge is B, one write followed by r reads therefore costs B(w+0.1r), versus B(1+r) without caching. Under these assumptions, the five-minute option saves prefix charges after one read; the one-hour option requires two. Paying for a longer lifetime is useful only if it enables enough valid reuse.
Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA
In the illustrated conversation, extending cache TTL bridges more request gaps and reduces repeated prefilling, at the cost of a larger retained working set.
Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA
In WEKA's reported comparison, an additional POSIX storage tier could hold reusable context but could not deliver it to GPUs fast enough to realize the benefit.
NVIDIA NVML: nvmlUtilization_t
NVML's GPU utilization is the percentage of a sampling interval during which at least one kernel executes. Its memory utilization is the percentage of that interval during which device memory is read or written. The documented sampling period depends on the product. These are activity-over-time measurements, not percentages of peak arithmetic throughput, allocated memory capacity, or successfully completed tasks.
Continuous batching — Transformers
Continuous batching schedules generation work at iteration granularity, allowing completed requests to leave and new requests to enter without waiting for every sequence in a fixed batch to finish. This makes variable sequence lengths important to serving utilization. Scheduling still has to respect available memory and latency constraints.
MIT OpenCourseWare: Review of Queueing Models
The queueing models distinguish arrival rate, service rate, utilization, waiting time, and total residence time. For k identical servers, utilization is arrival rate divided by their combined service rate. The single-server model with Poisson arrivals and general service times shows that greater service-time variance increases expected queue length and waiting. The general-arrival approximation additionally includes interarrival variability and a utilization term that grows sharply near saturation. Consequently, equal average work and equal nominal capacity can produce different waiting behavior.
Notes on Little's Law — Karl Sigman
Little's Law relates the long-run average number of items in a system to arrival rate multiplied by average residence time: L = lambda W. Sigman's sample-path formulation defines residence from entry to departure and states conditions requiring finite limiting arrival rate and average residence time. The chosen system can include waiting and service together or only one region. Consequently, concurrency, arrival rate and latency must refer to matching boundaries; average active execution cannot be substituted for all requests when residence time includes queueing.
John D. C. Little: Little's Law as Viewed on Its 50th Anniversary
Little's 2011 retrospective revisits his 1961 queueing result and develops finite-observation-window formulations. When the system starts and ends empty, average occupancy equals arrivals per unit time multiplied by average residence time exactly, without requiring stationary traffic. With work present at either boundary, his extension includes initially present items and counts only their residence within the observation window. Those quantities are not ordinary completed-request throughput and full completion latency. The paper distinguishes this retrospective accounting identity from forecasting and describes applications in operations management and computer architecture.
Optimizing inference for voice models in production
Once generation sustains real-time playback, prioritize startup latency and concurrency rather than maximizing tokens per second.
Google SRE: Handling Overload
Admission controls and per-customer quotas limit resource consumption so one workload does not exhaust shared capacity. Resource usage can be a better capacity signal than requests per second because requests vary in cost. Graceful degradation reduces work by returning less complete results or using cheaper, potentially stale cached data. Client-side throttling can prevent rejected requests from consuming backend resources. Under extreme overload, even degraded computation may be impossible and explicit errors are necessary.
Modal: Scaling Out
Modal documents separate controls for maximum containers, minimum warm containers, an active-service buffer, and idle scale-down duration. Maintaining a larger warm pool or buffer raises costs while reducing the chance that new inputs wait for container startup. A longer scale-down window can reduce repeated startup and shutdown for intermittent demand. Default function behavior scales to zero when there are no inputs.
Claude Platform: Rate limits
Claude documents separate request-per-minute, input-token-per-minute and output-token-per-minute limits. Short bursts can exceed enforcement intervals even when a minute-level average appears acceptable, and sharp usage increases can encounter acceleration limits. Limits are maximum permitted usage, not guaranteed service capacity. For most documented models, cache creation and uncached input count toward input-token limits while cache reads do not; specified exceptions exist. Consequently, losing cache reuse can increase both input charges and quota consumption, while a cache hit does not remove output-token or request constraints.
Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod
In this deployment demonstration, a returned request spent about 41 seconds queued but only about 1.5 seconds executing; the speaker attributes the longer wait partly to cold start.
Claude Platform: Batch processing
The Message Batches API processes submitted requests independently and asynchronously, with documented pricing at half standard API prices. Results become available when processing ends; unfinished batches expire after 24 hours. Limits apply separately to batch HTTP requests and queued requests, and processing can slow with demand. The documentation identifies evaluations, analysis, and bulk content generation as suitable examples when immediate responses are unnecessary. This is a concrete tradeoff between scheduling flexibility and charges, rather than a promise of faster interactive completion.
AI Engineering 201: Inference
Scale-to-zero inference suits intermittent workloads, but trades fine-grained scaling control for managed operations and low idle compute expense.
AI Engineering 201: Inference
The speaker's examples challenge the assumption that running open weights is automatically cheaper than buying API inference.
MLCommons: MLPerf Inference Rules
MLPerf defines a benchmark run around a specified system, preprocessing and postprocessing, workload scenario, latency requirement, and output-quality requirement. Its server scenario generates randomly timed queries using a Poisson process and measures the supported arrival rate under latency constraints; its offline scenario supplies all samples together. Accuracy is checked separately using the same implementation. The rules also restrict caching of queries and intermediate results, demonstrating that reuse permissions are part of a benchmark's workload contract.
Quantifying infrastructure noise in agentic coding evals
Anthropic varied resource allocation across Terminal-Bench configurations while holding the model, harness, and tasks constant. Resource enforcement changed infrastructure failures and task success. The report distinguishes additional headroom that reduces transient container failures from larger allocations that enable different solution strategies. In one example, installing a data-science dependency stack exhausted memory before solution code was written, whereas a leaner approach was possible. Thus an unsuccessful trajectory can reflect both an action strategy and the environment's constraints rather than a model-only defect.
Grafana k6: Arrival-rate VU allocation
An arrival-rate configuration expresses intended demand, but k6 still needs an available virtual user for each iteration. Longer task durations occupy users longer. When insufficient users remain, k6 records dropped_iterations for work it cannot start. Allocating additional users during execution also consumes CPU and memory and can distort measurements. Thus a valid load comparison must distinguish configured arrival rate, work actually started, and work omitted by the generator.
Grafana k6: Open and closed models
In a closed load model, each virtual user's next iteration waits for its previous iteration to finish. Slower responses therefore reduce the rate at which that test offers new work. Grafana identifies this coupling as coordinated omission when the intended experiment requires independently arriving demand. Open-model executors schedule iteration starts independently of completion. The published examples contrast these behaviors using requests that take approximately six seconds.
MLCommons: Introducing the MLPerf End-to-End RAG Inference Benchmark
The benchmark separates document ingestion from multi-hop question answering. A QnA task can repeatedly rewrite queries, retrieve, rerank, grade documents, and check evidence before answering. It reports tasks per second because token or hop rates do not represent the complete variable-length workflow. Performance runs use recorded intermediate inputs to fix retrieved documents and hop counts while still generating outputs; accuracy is evaluated separately. The initial release covers offline throughput, with all tasks available together.
RouteLLM: Learning to Route LLMs with Preference Data
RouteLLM selects a model from the query before seeing its generated response, unlike post-generation scoring or a sequential cascade. Its binary router predicts a preference outcome and applies a threshold controlling the fraction sent to the stronger model. Evaluation measures routed-answer quality alongside strong-model call share and varies the threshold to expose tradeoffs.
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
FrugalGPT formulates query answering as maximizing quality subject to an average spending constraint. Its cascade calls models sequentially, scores each generated answer, and stops when a stage's threshold is met. The budget constraint sums the costs of all models invoked before stopping, including earlier answers that were not selected. Thus escalation does not erase the expense of the cheaper first attempt.
Hacking the Inference Pareto Frontier
Evaluate combinations of techniques because a quality improvement that adds latency can be paired with a serving optimization that reduces it.
Google SRE Workbook: Canarying Releases
Canarying is a partial, time-limited deployment evaluated against a control before wider rollout. It requires a way to limit exposure, evaluate the candidate, and connect the evaluation to release decisions. The worked example separates metrics by version and pauses or rolls back when the candidate's error behavior diverges. Exposure duration and population size bound the service impact of a bad change.
RAG and the MongoDB Document Model
The illustrated LangChain semantic cache uses MongoDB to return an existing answer when the augmented prompt is semantically similar to a cached request.
AI Engineering 201: Inference
Interactive services should choose batching and caching around their actual bottleneck and latency budget; throughput-only jobs can favor larger batches.
Selective Question Answering under Domain Shift
Selective QA answers only when a confidence score exceeds a threshold. Coverage is answered_questions/all_questions; empirical selective risk is wrong_answers/answered_questions, undefined when none are answered. A risk–coverage curve varies the threshold; lower area and greater coverage at a fixed risk are desirable. The paper finds maximum answer probability overconfident on out-of-domain examples. A calibrator trained using in-domain and some known out-of-domain data achieved average coverage of 56.1% at 80% answered-question accuracy, versus 48.2% for maximum probability. Application inference: a verbal confidence statement does not establish calibration; evaluate correctness against confidence and risk–coverage on held-out deployment-like and shifted data.
Hacking the Inference Pareto Frontier
Route using both reusable prefix state and current worker load; maximizing KV match alone can increase queueing.
Hacking the Inference Pareto Frontier
A worker balance fitted to an initial workload can become unsuitable when the user mix changes; scale prefill and decode capacity with the evolving workload.
How fast are LLM inference engines anyway?
The demonstrated benchmark interface selects throughput results under a time-to-first-token requirement rather than treating throughput alone as sufficient.
The State of Model Routing — NVIDIA, Cognition, OpenRouter
Devin Fusion is described as retaining a frontier agent for planning and supervision while delegating implementation to cheaper models.
The State of Model Routing — NVIDIA, Cognition, OpenRouter
Keep most raw context with one model, send compact summaries or file references to collaborators, and preserve retrievable originals outside the context window.