Part I — From request to generated tokens
The serving contract
A serving request is not merely the user's visible text. It is the complete model-visible sequence produced by a model-specific chat template, together with generation controls and an allowance for new output. System instructions, message history, retrieved passages, tool definitions, role markers, and special tokens may all contribute to the input. The application should count the complete serialized request and reserve room for generation before admission. A context limit constrains the combined sequence; cache capacity does not enlarge that architectural limit.
From acceptance to release
The request then passes through distinct responsibilities. Admission control decides whether bounded resources and policy permit activation. An admitted request may wait in a queue before model work begins. Execution produces zero or more incremental outputs until an end token, stop match, output limit, cancellation, or error supplies a disposition. Completion initiates cleanup, but it does not prove that already-launched accelerator work stopped instantly or that every allocation became reusable at the same moment. One documented vLLM lifecycle, for example, can defer block release while a GPU step or remote cache transfer remains in flight.
Three operational quantities
| Quantity | Meaning | Required boundary |
|---|---|---|
| Latency | Elapsed time for one named interval | Start event, end event, and observation location |
| Throughput | Completed work per unit time | Work unit, completion rule, and measurement duration |
| Capacity | Workload sustainably served under stated constraints | Arrival pattern, request mix, latency objectives, and completion requirements |
These quantities answer different questions. High accelerator utilization says that a resource was busy, not that useful requests completed within their objectives. Likewise, an HTTP success status describes a transport or API boundary; it does not establish answer quality. Production inference also needs readiness, lifecycle management, observability, recovery, and controls around the model engine itself. The chronology traces distinct serving constraints: latency-aware batching, benchmark contracts, iteration scheduling, speculative execution and KV organization. These mechanisms can coexist.
Different constraints, complementary serving mechanisms
March 2017ClipperAdapt batch size to online prediction’s latency objective.
Contributors: Berkeley team
What changed: Published at NSDI 2017. Per-model controllers adjusted maximum batch size to amortize overhead without ignoring interactive latency. Clipper addressed ordinary prediction calls, not token-by-token generation.
November 2019 preprint; ISCA 2020MLPerf InferenceCompare performance under explicit workload, quality, and latency contracts.
Contributors: Reddi and 46 coauthors
What changed: The initial server scenario measured sustainable throughput under independent arrivals and a tail-latency constraint. The offline scenario supplied all inputs together. Their distinction explains why bulk throughput does not establish interactive capacity.
July 2022OrcaReturn scheduling control after each generation iteration.
Contributors: Seoul National University and FriendliAI researchers
What changed: Published at OSDI 2022. Completed requests could leave and new requests enter between iterations, addressing wasted work and delayed admission when outputs have unequal lengths.
November 2022 preprint; ICML 2023Fast Inference from Transformers via Speculative DecodingTrade draft and verification work for fewer serial target steps.
Contributors: Google
What changed: The procedure preserves the target output distribution while using draft proposals and parallel target scoring. Benefit depends on accepted advancement, draft cost, and wasted verification work—not acceptance rate alone.
September 2023 preprintPagedAttentionOrganize KV state in noncontiguous blocks allocated as needed.
Contributors: Woosuk Kwon and colleagues
What changed: Block-based organization reduces reservation waste and fragmentation and permits compatible state sharing. It addresses memory organization, separately from the serial-execution constraint targeted by speculative decoding.
Prefill and decode
An autoregressive language model maps its current prefix to vocabulary scores called logits. A selection rule chooses one token, appends it to the prefix, and changes the conditional distribution used for the following choice. The full model mechanism lives in Producing an unknown continuation; the serving consequence is that known input positions and unknown output positions create different execution shapes.
The first-token boundary
Prefill processes the complete supplied prefix and establishes contextual state for its positions. The logits at the final prompt position support selection of the first output token. Because all prompt tokens are already known, an implementation can process substantial work across their positions together while causal masking prevents a position from reading future content. Decode begins after that first selection. Each iteration processes the newest selected token with retained state, produces the next logits, selects another token, and repeats. The next iteration cannot know its input token until the preceding selection finishes.
A selected token gets its KV state in the next step
Symbolic prompt P₁–P₃; ordinary full-history cache. Columns follow computation dependencies from left to right.
| At this boundary | Prefill → select O₁ | Process O₁ → select O₂ | Process O₂ → select O₃ |
|---|---|---|---|
| Known input processed | P₁ P₂ P₃, together under causal masking | O₁, selected in the prior column | O₂, selected in the prior column |
| Retained numerical KV read | No prior prompt cache in this example | KV(P₁ P₂ P₃) | KV(P₁ P₂ P₃ O₁) |
| New numerical KV written | KV(P₁ P₂ P₃) | KV(O₁) | KV(O₂) |
| Logits used to select | Final prompt position P₃ → O₁ | Processed O₁ position → O₂ | Processed O₂ position → O₃ |
| Cache after selection | 3 positions: P₁ P₂ P₃ | 4 positions: P₁ P₂ P₃ O₁ | 5 positions: P₁ P₂ P₃ O₁ O₂ |
| Selected, not yet processed | O₁ has no KV yet | O₂ has no KV yet | O₃ has no KV yet |
Prompt length therefore primarily changes prefill work and the amount of state created before generation. Output length primarily changes the number of decode iterations and how long each active sequence retains state. Prefill often exposes matrix-shaped parallel work, whereas low-batch decode often performs less arithmetic per byte moved. These are tendencies, not universal bottleneck labels: batching, architecture, kernels, cache representation, hardware, and sequence lengths can move either phase into another regime. The correct diagnosis comes from profiling the intended workload.
Part II — State and progress
The KV cache
Attention separates addressing from transmitted content through keys and values. During prefill, each transformer layer computes key and value representations for the supplied positions. A KV cache retains the positions selected by the model architecture and cache policy. In an ordinary full-history cache, that means every processed position; sliding-window or chunked layers may retain only a bounded subset. During decode, the new query can attend to retained keys and combine the corresponding values without recomputing their old projections, then append state for the newly processed position. The cache stores numerical representations derived from tokens—not text, an answer cache, or durable model memory.
A cache-payload estimate
The number of KV heads is architectural. Conventional multi-head attention may retain a key and value head for every query head. Grouped-query attention shares each key-value pair among a group of query heads, while multi-query attention uses one key and one value head for all query heads. That can reduce cache storage and bandwidth, but changing the head arrangement changes the model architecture; it is not a lossless allocator setting applied after deployment.
Keep the memory ledger separated
| Category | Lifetime | Why it matters |
|---|---|---|
| Model weights | Usually resident | Must fit before request state; precision affects bytes and supported arithmetic |
| KV state | Request or reusable-prefix lifetime | Grows with retained positions and active sequences |
| Activations and workspace | Transient peaks | Can reduce the cache budget even though allocations are temporary |
| Runtime and graph storage | Engine-dependent | May be reserved separately from weights and KV payload |
| Free or fragmented capacity | Changes over time | Determines whether another request or block can be admitted |
PagedAttention, introduced by Woosuk Kwon and colleagues in a September 2023 preprint, organizes KV state in fixed-size noncontiguous blocks allocated as needed. This reduces reservation waste and fragmentation and permits compatible sharing. Prefix caching extends reuse across requests, but only for exactly compatible preceding state. A practical cache identity can include parent-block identity, exact token IDs, adapter identity, multimodal hashes, and an isolation salt. Similar meaning or an identical suffix is insufficient. Reuse saves prefill work; it does not accelerate the new output tokens. In a documented vLLM implementation, cache blocks carry reference counts: releasing one request decreases ownership, but a block referenced elsewhere cannot be overwritten. A zero-reference cached block may remain reusable until eviction. Other engines may use different ownership and eviction contracts.
Latency has boundaries
“Latency” is incomplete without a start event, end event, and observation location. Queue time ends when execution starts. Prefill duration covers prompt processing at the engine. Server time to first token reaches the server's first token event, while a client-observed time to first token (TTFT) commonly ends when the client receives the first nonempty content response. Network, proxy, decoding, and buffering can separate those events. Observability's elapsed-time framework develops this boundary discipline for complete systems.
Progress and completion metrics
| Metric | Typical observation | Important qualification |
|---|---|---|
| Queue time | Admission or arrival to engine start | Requires a defined admission boundary |
| TTFT | Request send to first nonempty content received | Includes client-visible path; server TTFT may differ |
| ITL | Intervals between successive responses or chunks | A chunk can contain multiple tokens |
| TPOT | Post-first-token generation time divided across later output tokens | Some tools label a related average as ITL |
| Request latency | Request start to final content response | Does not necessarily include later cleanup |
| Output-token throughput | Generated tokens divided by benchmark duration | Tokenizer, completion accounting, and workload must match |
A generated token, a decoded text fragment, a transport chunk, and a rendered UI update are different events. A streamer may buffer token IDs until they form printable text, and transport infrastructure may combine fragments. Incremental delivery can make an interface responsive before the full request completes, but it need not reduce model execution time. A delivered prefix also does not prove eventual successful completion.
One request, several latency intervals
Example timingsServer work, client visibility, completion, and cleanup use different start and end events.
Read the diagram as text
- Client request lifetime. Send through final client content. 0 to 100 ms; duration 100 ms.
- Queue residence. Arrival until engine work begins. 0 to 18 ms; duration 18 ms. Parent: Client request lifetime.
- Prefill. Prompt processing before first token selection. 18 to 43 ms; duration 25 ms. Parent: Client request lifetime.
- First-content transit. Server selection and delivery to first nonempty client content. 43 to 51 ms; duration 8 ms. Parent: Client request lifetime.
- Decode execution. Repeated generation after the first selection. 43 to 94 ms; duration 51 ms. Parent: Client request lifetime.
- Incremental delivery. Decoded fragments transported while generation continues. 51 to 100 ms; duration 49 ms. Parent: Client request lifetime.
- Deferred cleanup. Resource ownership is released after terminal state permits it. 94 to 108 ms; duration 14 ms.
Report distributions rather than only means. The 99th percentile is the value below which 99 percent of the specified observations fall; it is not an average. Partitioning by prompt length, output length, and request class often explains more than one aggregate percentile. Do not add independently computed p99 values for queue, prefill, network, and decode and call the sum an end-to-end p99: those percentiles may come from different requests.
Part III — Sharing a server
Continuous batching
A batch executes work from several requests together. Larger parallel operations can improve hardware use and amortize launch, scheduling, and weight-access overhead. But waiting to form the batch adds latency, and every active sequence consumes state. The useful batch is therefore constrained by the application's response budget and memory—not simply by the engine's configured maximum. Clipper, published at NSDI in March 2017 by a Berkeley team, adapted per-model batch limits to prediction-latency objectives. Its grouped ordinary prediction calls amortized overhead; it did not schedule autoregressive tokens.
Three batching policies
| Policy | Admission point | Characteristic tradeoff |
|---|---|---|
| Fixed batch | Before the batch starts | Simple, but unequal completion lengths can leave lanes idle |
| Dynamic batch | After a short formation window | Can increase batch size by purchasing extra waiting time |
| Continuous or iteration-level batch | Between generation iterations | Completed requests leave and new requests enter while others continue |
Orca, published at OSDI in July 2022 by Seoul National University and FriendliAI researchers, returned scheduling control after each transformer iteration. Completed requests could leave and new work enter. Continuous batching is especially useful because generated outputs have unequal and initially unknown lengths. When one request ends, the scheduler can replace it without waiting for every other sequence in the batch. This reduces idle lanes, but it does not eliminate interference: new prefills compete with ongoing decodes, larger active sets require more KV state, and each request's token cadence can slow even as total device token throughput rises. Distinguish the configured sequence limit, the currently active sequences, and the prompt or decode tokens actually scheduled in an iteration.
Scheduling and admission
Admission control and scheduling solve different problems. Admission limits how much work can enter the bounded service; the scheduler chooses which admitted work runs next. If offered work persistently exceeds service capacity, rearranging the queue cannot make every objective feasible. The service must reject, defer, shed, degrade, or add capacity.
Allocating delay
A long prefill can delay token production for existing requests. Conversely, always prioritizing decode can make new prompt-heavy requests wait indefinitely for their first token. Chunked prefill divides a long prompt into bounded pieces that the scheduler can interleave with decode. Smaller chunks reduce individual stalls but introduce more scheduling boundaries, fixed overhead, and repeated reads of state from earlier prompt chunks. The best token budget is workload- and hardware-dependent.
Smaller prompt chunks divide delay—and add work
A and B are already decoding at time 0 and need 5 and 2 more steps. At 2ms, C arrives with a 12-token prompt and D with a 2-token prompt. Each new request produces one token at prefill completion, then needs 2 decode steps. Three active slots; D replaces B at the next iteration boundary. Memory is sufficient.
| Quantity | Unchunked (12) | Selected (4) |
|---|---|---|
| C arrival → first selection | 10ms | 15.2ms |
| Largest A/B engine selection gap | 9ms | 5ms |
| Finish all work | 23ms | 26.2ms |
| Iterations / overhead | 5 / 5ms | 7 / 7ms |
| Prompt tokens / decode steps | 14 / 11 | 14 / 11 |
| Earlier prompt positions reread | 0 | 12 |
Fixed teaching workload and cost model
One iteration costs 1ms scheduling overhead + 0.5ms per prompt token + 1ms per decode step + 0.1ms per earlier prompt position read when continuing a prefill. Prompt requests share the selected prompt budget in arrival order (C before D for their tied arrival); every active decoding request gets one step. The total batch-token limit is fixed at 15. Admission and completion changes occur only between iterations. Prior A/B prefill is outside this window. No transport costs, memory pressure or failures are modeled.
Unchunked baseline
| Iteration | Time (ms) | A | B | C | D |
|---|---|---|---|---|---|
| 1 | 0–3 | D1 | D1 | — | — |
| 2 | 3–12 | D1 | D1 · done | P12 | — |
| 3 | 12–16 | D1 | — | D1 | P2 |
| 4 | 16–20 | D1 | — | D1 · done | D1 |
| 5 | 20–23 | D1 · done | — | — | D1 · done |
Prompt budget 4 / iteration
| Iteration | Time (ms) | A | B | C | D |
|---|---|---|---|---|---|
| 1 | 0–3 | D1 | D1 | — | — |
| 2 | 3–8 | D1 | D1 · done | P4 | — |
| 3 | 8–12.4 | D1 | — | P4 | Waiting |
| 4 | 12.4–17.2 | D1 | — | P4 | Waiting |
| 5 | 17.2–21.2 | D1 · done | — | D1 | P2 |
| 6 | 21.2–24.2 | — | — | D1 · done | D1 |
| 7 | 24.2–26.2 | — | — | — | D1 · done |
P = prompt tokens processed; D1 = one decode step; done = leaves before the next iteration. Gaps are engine selections, not measured client ITL.
Policy dimensions
| Decision | Benefit sought | Failure to test |
|---|---|---|
| First-come-first-served | Simple ordering | A long request blocks short later work |
| Priority classes | Protect important traffic | Lower classes starve |
| Aging or SLO-relative urgency | Bound starvation across classes | Sustained overload still violates objectives |
| Preemption | Let urgent work run sooner | Retained KV uses memory; swapping or recomputation costs work |
| Cancellation | Stop unwanted lifecycle work | Propagation and resource release may be delayed |
| Backpressure or admission limits | Bound queued and active work | Rejection or deferral becomes caller-visible |
Prefix-aware routing creates another tension. Sending a request to a worker with reusable state can avoid prefill, but routing only by cache match can overload that worker and turn saved computation into queueing. A useful policy considers both compatible reusable state and present load. In large deployments, separating prefill and decode workers can reduce phase interference, but transferring KV state adds communication whose cost depends on its size, placement, and bandwidth.
Part IV — Selecting and delivering output
Sampling, stopping, and streaming
Each decode step begins with logits. Greedy decoding chooses a highest-scoring token. Sampling converts processed scores into a probability distribution and draws from it. Temperature changes concentration; lower values emphasize larger logits while higher values flatten the distribution. Top-k keeps a fixed number of high-probability candidates. Top-p, or nucleus sampling, keeps the smallest high-probability set whose cumulative mass reaches a threshold, then renormalizes. These settings alter token selection, not whether the answer is true.
Selection is not system determinism
Enabled controls also have an implementation order. One documented Transformers version applies temperature, then top-k, then top-p before softmax and sampling; custom processors or other engines may differ. Greedy execution instead takes an argmax. A random seed constrains random-number generation, but portable reproducibility can additionally require the same engine version, hardware, scheduling behavior, and supported batch-invariant execution. Temperature zero does not validate reasoning or guarantee identical outputs across changed environments.
Ways generation ends
After selecting a token, the server appends it and checks termination. An end-of-sequence token is model-visible. A stop string is matched against decoded text and may require holding back a suffix so the stop marker is not leaked. A maximum-output limit ends generation because a budget was exhausted, not because the task necessarily completed. Structured or machine-consumed responses must also handle refusal and incomplete output rather than parsing every partial prefix as a finished record.
Streaming exposes completed fragments while generation continues. It can reduce perceived waiting without changing the number of model steps. Backpressure at the transport layer prevents an unbounded application buffer, but pausing a network writer does not automatically pause accelerator generation; that requires explicit propagation into the serving scheduler.
Speculative decoding
Ordinary autoregressive generation pays for one serial target-model step per advancement. Speculative decoding uses a cheaper draft process to propose several future tokens, then asks the target model to score their conditional distributions together. The target—not the draft—determines what is committed. This can exchange extra arithmetic for fewer serial target invocations, which is useful when target execution has spare parallel capacity but repeated state or weight access dominates elapsed time. Google's Fast Inference from Transformers via Speculative Decoding appeared as a November 2022 preprint and at ICML 2023. Its analysis separates accepted advancement from draft cost and wasted verification work.
Accept, reject, correct
In distribution-preserving speculative sampling, proposals are examined left to right. Proposal , drawn from draft distribution , is accepted with probability , where is the target distribution for that position. The accepted prefix is retained. At the first rejection, that token and later proposals are discarded; a replacement is drawn from the normalized positive part of . If every proposal passes, the target supplies an additional token. This procedure preserves the target sampling distribution within numerical precision; it does not guarantee identical samples or correct answers.
Unused proposals must not leak into output or persistent target state. In Transformers v4.50.0, assisted generation supplies only the valid continuation to the streamer and crops target KV to the total committed sequence length minus one. The last selected token has not yet been processed; after a rejection, this can be the replacement token. Logical cropping need not release allocator reservations immediately. Useful advancement must justify both draft and verification cost.
Commit only the accepted prefix and its correction
At each position, accept xᵢ with probability min(1, pᵢ(xᵢ)/qᵢ(xᵢ)). Examine proposals left to right and stop at the first rejection.
| Main branch | Position 1 | Position 2 | Position 3 | Position 4 |
|---|---|---|---|---|
| Draft proposals | x₁ from q₁ | x₂ from q₂ | x₃ from q₃ | x₄ from q₄ |
| Target conditional scoring | p₁(· | prefix) | p₂(· | prefix,x₁) | p₃(· | prefix,x₁,x₂) | p₄(· | prefix,x₁,x₂,x₃) |
| Acceptance | ✓ Accept x₁ | ✓ Accept x₂ | × Reject x₃ | — Not tested; discard x₄ |
| Committed continuation | x₁ | x₂ | r: replacement | None |
| Valid tokens supplied to streamer | x₁ | x₂ | r | None |
| Retained new target KV | KV(x₁) | KV(x₂) | No KV(r) yet; drop KV(x₃) | Drop KV(x₄) |
Replacement r ∼ normalize(max(p₃ − q₃, 0)), not an ordinary draw from p₃.
Conditions governing benefit
Tune speculative decoding for the deployed target, draft process, request content, and concurrency regime. Sweep draft length and related settings across relevant batch sizes, then compare elapsed time and accepted advancement—not acceptance rate alone. A configuration that helps one workload can lose its benefit when drafting or verification competes with other requests.
Part V — Development and capacity
Serving developments
A serving design combines choices with different effects. Batch formation controls which work runs together; iteration scheduling controls when membership changes; state organization controls what occupies memory. No single setting resolves all three constraints.
Saved computation can become additional waiting. A larger batch or a useful prefix-cache hit matters only if the complete request still meets its delivery objective.
More retained state can avoid repeated work while reducing room for other sequences. Recomputation can therefore remain useful for short contexts or severe memory pressure.
Speculation changes how many tokens advance per target invocation. It still competes for arithmetic, memory and scheduling time with other requests, so a single-request benefit need not survive concurrent load.
Evaluate the combined policy under a matching workload and completion contract. Preserve the distinction between engine work, client-visible progress and safe resource reuse when interpreting a change.
Precision and resources
Numerical precision is a serving resource choice applied separately to weights, activations, and KV state. Lower-bit weight storage can reduce resident model bytes, but storage precision, computation precision, and device placement are independent. Some methods preserve higher precision for outliers; offloaded weights may use a wider format. Halving weight bytes therefore does not halve total serving memory or latency.
Cache precision
KV quantization targets a different allocation. KIVI, a February 2024 preprint later accepted at ICML 2024, used asymmetric two-bit quantization with a full-precision recent window. Its reported single-A100 experiment showed how reduced KV storage could permit larger batches and greater throughput, while quality varied by model, task, bit width, group size, and residual-window length. Fewer stored bytes can also add conversion overhead, so capacity gains do not independently establish latency gains.
Realized speed requires compatible arithmetic and kernels. In PyTorch's reported 2023 GPT-fast experiment, compilation and a static KV cache first exposed weight-loading bandwidth as the next constraint; supported int8 weight storage then improved the reported batch-one generation rate further. The point is bottleneck migration, not the historical numbers: changing coordination can expose bandwidth, and changing bytes can expose another limit. The numerical mechanisms of calibration, scales, outliers, and quantization-aware training belong in Quantization.
A comparison ledger
| Field | Why it must be reported |
|---|---|
| Weight format and bytes | Shows resident-model change |
| KV format and bytes per retained position | Shows request-state change |
| Activation and workspace peak | Prevents hidden capacity changes |
| Supported hardware and kernels | Establishes whether lower precision accelerates execution |
| Latency and sustainable capacity | Measures operational consequence |
| Quality and completion behavior | Detects altered outputs rather than assuming equivalence |
The capacity envelope
Capacity is an operating region, not a hardware label. Resident weights and runtime allocations consume memory before requests arrive. Each prompt adds prefill work and initial KV state. Each generated token adds another decode iteration and usually another retained position. Concurrency can improve reuse and batching while increasing contention and state. Arrival rate adds queueing when offered work approaches what the complete service can finish.
A consistency relation
Near saturation, completed throughput may flatten while queue residence and tail latency rise. Output tokens per second, requests per second, per-request token pace, and concurrent sequences therefore cannot substitute for one another. Prompt-heavy work emphasizes first-token delay and prompt processing; generation-heavy work spends more iterations retaining state; interactive service constrains onset and cadence; offline work can accept waiting to maximize completed volume.
Workload objectives
| Workload | Primary concern | Common constraint to test |
|---|---|---|
| Interactive chat | TTFT and output cadence | Batch formation, prompt interference, network delivery |
| Long generation | Completion time and state residency | Decode service rate and KV growth |
| Prompt-heavy processing | First-token latency and prefill capacity | Prompt length, chunking, and prefix reuse |
| Offline batch | Completed volume and total job duration | Largest efficient batch under memory limits |
| Streaming voice | Useful response onset and concurrent streams | Entire audio pipeline, not text token rate alone |
More memory can raise the number or length of active sequences without increasing a compute-limited token service rate. Conversely, faster arithmetic may leave capacity unchanged when KV storage or memory bandwidth is limiting. Admission policy must therefore reserve against both state and work. During severe overload, explicit rejection or a clearly exposed degraded mode is safer than silently accepting requests that cannot meet their contract.
Choose capacity within both latency objectives
Filled circle: qualifies; larger circle: selected. Empty circle: misses an objective. Cross: invalid run. Dashed: chosen objective. L14 has unavailable measurements and is not plotted.
| Offered /s | Successful / 60s | Completed /s | p99 TTFT | p95 TPOT | Run status / errors | Decision |
|---|---|---|---|---|---|---|
| 2 | 120 | 2 | 80ms | 12ms/token | stable / 0 | Qualifies |
| 4 | 240 | 4 | 140ms | 20ms/token | stable / 0 | Qualifies |
| 6 | 360 | 6 | 260ms | 32ms/token | stable / 0 | Selected |
| 8 | 480 | 8 | 520ms | 48ms/token | stable / 0 | Misses objective |
| 10 | 540 | 9 | 1000ms | 68ms/token | unstable / 0 | Invalid run |
| 12 | 660 | 11 | 1400ms | 90ms/token | errored / 60 | Invalid run |
| 14 | Unavailable | Unavailable | Unavailable | Unavailable | incomplete / unknown | Invalid run |
Dataset and measurement contract
Fictional sweep of one unchanged configuration and workload: 128-token prompts, 32-token outputs, periodic independent arrivals. Discard 30s warmup, then observe 60s. Throughput counts successful final-content completions inside that window divided by 60s. Every counted output passes the fixed task-quality check; errors are separate. p99 TTFT is over completed requests, from client send to first nonempty content. For each completed request, TPOT = (final-content time − first-content time) / 31; p95 is taken across those per-request averages. Valid runs require complete logs, zero errors and no positive queue-length trend over the final 30s. L10 is unstable, L12 has 60 errors, L14 has incomplete logs. Tied throughput selects the lower offered rate, then lexical run ID.
Part VI — Measurement and decisions
Benchmark the workload
MLPerf Inference, introduced by Reddi and collaborators in a November 2019 preprint and presented at ISCA 2020, distinguished independently arriving server traffic under tail-latency limits from offline bulk throughput, with required output-quality targets. A serving benchmark is defined by its workload and execution contract, not just a model and accelerator. Record the exact model revision, tokenizer, chat template and complete serialization; hardware and topology; engine and runtime versions; weight and KV formats; scheduler, batch-token limits, and cache policy; sampling and stopping controls; prompt and output distributions; warmup and readiness; arrival process or concurrency; duration; observation boundaries; errors; completion accounting; and quality checks. Use completed-work timing and sweep the important workload range.
Open-loop and closed-loop load
| Load model | Issuance rule | Question answered | Failure to expose |
|---|---|---|---|
| Open loop | Arrivals occur independently of prior completions | Can the service sustain an offered arrival rate under latency constraints? | Delayed issuance and unfinished work must remain visible |
| Closed loop | A client issues new work after earlier work completes while holding concurrency | What throughput and latency occur with this many active clients? | A saturated load generator can silently cap offered demand |
Separate unloaded single-request latency, bulk throughput, and the load sweep between them. Fixed input and output lengths can isolate execution behavior but do not represent production unless the real distribution is similarly fixed. Continuing beyond natural stop tokens controls length for a benchmark while changing ordinary completion behavior; disclose it. Likewise, raw single-batch engine throughput is not a serving result with in-flight batching, queueing, and transport.
The client and network are part of an endpoint measurement. Record client location, connection behavior, request-generation capacity, target and achieved arrival rates, and incomplete work. A benchmark client that cannot feed the server can make a fast deployment appear slow; a distant or buffered transport can dominate caller-visible TTFT even when model execution is fast. Match output length, completion status, and quality so shorter, refused, or failed responses are not counted as optimizations.
Locate the bottleneck
Diagnosis begins with a symptom conditioned on workload. Align queue depth, request lengths, batch composition, KV occupancy, preemptions, prefill and decode timing, accelerator activity, host gaps, transfers, client observations, cancellations, errors, and completion outcomes. A high utilization percentage does not identify whether compute units, memory bandwidth, host coordination, or another resource limits useful work.
Controlled sweeps
| Change one variable | Observation | Supported hypothesis to investigate |
|---|---|---|
| Increase prompt length | TTFT and prefill time rise; TPOT stable | Prompt processing is consequential |
| Increase output length | Completion and KV residency rise | Decode duration or cache growth matters |
| Increase offered rate | Queue rises; completed throughput flattens | Service is near sustainable capacity |
| Increase active sequences | Aggregate throughput rises but TPOT worsens | Batching helps throughput while adding interference |
| Reduce cache budget | Preemption or admission failures increase | KV capacity constrains concurrency |
| Change precision | Bytes fall but latency does not | Kernel support or another bottleneck dominates |
| Change speculative settings | Acceptance rises without elapsed-time gain | Draft or verification overhead dominates |
Expect bottleneck migration: relieving one constraint can make another resource determine useful work. Reducing resident weight bytes may permit more active sequences until KV capacity, scheduling, compute, or host coordination dominates. An external cache tier can add capacity yet stall accelerators if its reads and writes cannot keep pace. State what changed, what remained fixed, which resource signal moved, and whether the completion and quality contract still held.
Warm execution is not startup
Cold starts form another path. New capacity may require resource acquisition, container startup, transfer of model weights through host memory into accelerator memory, compilation or graph capture, and warmup before readiness. A system can have excellent warm-request latency yet fail to absorb a burst because replicas become useful too slowly. Measure startup stages separately from steady-state inference and test actual scaling events rather than inferring them from loaded-instance throughput.
Routing and reasoning budgets
Three adjacent responsibilities change the workload in different ways. A model-routing or gateway policy chooses a model or provider, escalation path, and failure fallback. A reasoning and test-time-compute policy chooses how many samples, search steps, verifications, or retries to request in pursuit of a better solution. Inference serving executes and measures every invocation those policies create.
Responsibility boundary
| Responsibility | Decision | Output | Evaluate by |
|---|---|---|---|
| Routing | Where should this invocation run? | Selected model, provider, or fallback | Policy quality, compatibility, reliability, and total cost |
| Reasoning policy | How much solution work should be requested? | Samples, search, verification, and stopping decisions | Task quality under a stated compute and latency budget |
| Inference serving | How should each invocation execute? | Tokens, stream events, completion disposition, and resource usage | Latency, throughput, capacity, correctness, and operational reliability |
The boundaries compose but should not be collapsed. Re-querying a smaller model may be preferable to one larger-model call for a particular quality target, yet those repeated calls still consume requests, tokens, cache, and concurrency. Moving a predictable re-query into a router can remove client round trips and expose scheduling opportunities, but the quality gain belongs to the reasoning policy and the execution gain belongs to serving. Evaluate the resulting combined workload at matched task quality rather than describing additional computation as free.
Open questions
Can generative-serving systems expose provider-neutral, observable state transitions for cancellation and resource reuse? A useful contract would let conformance tests distinguish a terminal request from accelerator work or allocations that remain in flight.
How should schedulers optimize jointly for TTFT, token cadence, fairness, energy, and completed task quality under changing request distributions? Present policies usually prioritize only a subset, while sustained overload can make the objectives mutually infeasible. Progress would require workload-replay studies with explicit admission and degradation semantics, not only average throughput gains.
When does KV compression improve end-to-end capacity after accounting for conversion work, quality changes, allocator behavior, and the batch sizes it enables? Results vary across models, tasks, formats, and recent-window policies. Progress would be reproducible, matched-quality load sweeps that report payload bytes, actual allocations, preemption, TTFT, TPOT, and completed throughput.
Can speculative decoding remain beneficial under high continuous-batch pressure, where draft work and additional state compete with unrelated requests? Single-request acceptance rates do not answer this fleet-level question. Progress would connect accepted advancement to queueing, memory, fairness, and goodput across real prompt and output distributions.
What bounded reproducibility contract is practical across engine upgrades, kernel changes, hardware revisions, and different batch compositions? Seeds control only part of execution, while exact outputs may not be necessary for every application. Progress would define useful levels—from distributional equivalence to stable state transitions—and publish the conditions each engine can enforce.
