Purpose and development
What a routing layer does
A large language model, or LLM, generates responses from supplied inputs. A provider makes a model available through a serving endpoint. Choosing the model concerns its behavior; choosing the endpoint also concerns supported features, capacity, credentials, and operating conditions. Inference Engineering explains the complementary responsibility: executing the invocations that these choices create.
An LLM gateway mediates model calls; a router chooses their destination. A request policy governs both. Mediation alone does not predict answer quality.
| Responsibility | Decision input | Result |
|---|---|---|
| Gateway mediation | Request, identity, interface and policy | A controlled provider interaction |
| Model selection | Task information available before generation | A model chosen for the request |
| Quality escalation | A generated answer and its assessment | Accept it, seek another answer, or defer |
| Failure fallback | An execution failure and eligible alternatives | Another route attempted under bounded recovery |
Shared mediation is useful when several applications need common access controls, provider credentials, and request records. In the Aperture demonstration, the gateway holds provider keys outside an agent sandbox and uses network identity to govern access. That reduces reusable provider secrets in clients, while concentrating authority and introducing another service dependency. The gateway sees traffic passing through it, not every local action or external effect.
Direct integration remains a reasonable design when one application can enforce its requirements without a shared intermediary. Add mediation for concrete benefits that justify its processing overhead and operating burden. A request-management service is also narrower than AI Platform Engineering, which addresses shared capabilities and their organizational ownership.
Origins and turning points
Modern routing brings together two continuing lines of work. Network intermediaries centralize access to services. Algorithm selection chooses different computation for different inputs. Their combination matters because model services can differ both operationally and in the answers they produce.
| Development | Contribution |
|---|---|
| Rice's algorithm-selection report — July 1975 | John R. Rice formalized selecting algorithms using problem-instance features and performance criteria. Original report. |
| HTTP/1.1 intermediary roles — January 1997 | RFC 2068 distinguished proxies acting for clients from gateways presenting an entry point to another server. Historical specification. |
| Amazon API Gateway — July 9, 2015 | AWS's managed service combined authorization, traffic management, monitoring, and API version management ahead of backend applications. Announcement. |
| FrugalGPT — May 9, 2023 | Lingjiao Chen, Matei Zaharia, and James Zou developed answer-scored LLM cascades under an average API-cost constraint. Paper. |
| RouteLLM — June 26, 2024 | Isaac Ong and colleagues introduced preference-trained routing that chooses a model from the query before generating its answer. Initial preprint. |
The older selection problem already contained a lasting limitation: extracted features discard information, so identical features need not imply identical algorithm performance. LLM routers inherit that prediction problem. Gateways solve a different problem by mediating access and preserving interfaces. Neither line makes the other obsolete, and chronological succession alone does not establish direct influence.
Interfaces and eligibility
Preserve the request contract
Define the application's obligations before choosing an adapter. A logical request is one application request with one intended outcome. A provider attempt is one execution toward that outcome. Retrying or changing models creates another attempt; it should not silently create a new deadline, spending allowance, or definition of success. This applies the broader principle of explicit interface obligations.
| Contract element | Meaning |
|---|---|
| Input | Messages, attachments, and references whose meaning must survive translation. |
| Selection and output requirements | Requested model or policy alias; required modalities, structure, and tool behavior. |
| Execution limits | Generation allowance, overall deadline, and cancellation behavior. |
| Trusted context | Authenticated caller and operator policy, obtained independently of request text. |
| Correlation | Logical request identity retained across distinct attempt identities. |
An adapter translates between interfaces. A normalized envelope simplifies application code, but provider-specific behavior sometimes needs explicit extensions. Bedrock Converse, for example, combines common fields with model-specific additions and separately returns output, stop reason, usage, and metrics. Normalization should preserve those distinctions instead of collapsing them into a single success flag.
The response contract should retain content, proposed tool calls and their identities, destination information when available, completion disposition, and whether usage is known. A tool call is a proposed operation, not proof of execution. Streaming delivers incremental content before the terminal outcome is known; individual chunks are not completed answers. Preserve refusal, interruption, output-limit termination, and ordinary completion as distinguishable outcomes.
Check the concrete service path
Provider capability means supported behavior along a concrete path: model, endpoint, API version, adapter, and configuration. A matching JSON shape establishes much less. Anthropic's documented OpenAI SDK compatibility layer, for example, ignores response_format and function-calling strictness, strips audio input, and combines system/developer messages. Those limitations concern that compatibility layer, not Claude's native interface.
Check each required behavior separately: support for one feature cannot compensate for another missing feature. A schema describes the permitted structure of an output. In this example, the request requires image input, particular schema features, and a particular tool-call mode. Each endpoint's support is recorded as supported, unsupported, or unknown.
| Requirement | Endpoint A | Endpoint B |
|---|---|---|
| Image input | Supported | Supported |
| Required schema features | Supported | Unknown |
| Required tool mode | Unsupported | Supported |
| Eligibility | Excluded for this request | Unresolved until schema support is established |
Neither endpoint currently qualifies: A lacks the required tool mode, while B's schema support still needs verification. Reject an incompatible request or negotiate a changed requirement explicitly; silently dropping a field changes the contract. Providers can support different subsets of schema features. Structured Outputs and Tool Calling explains how to test that coverage.
Capacity and references also belong in compatibility checks. Tokens are model-specific representation units, so changing models can change the input count; count the complete request and reserve room for generation. An opaque file identifier is different from its content. Claude's Files API returns workspace-scoped identifiers and requires applications to maintain user-to-file authorization. Copying an identifier to another route neither transfers the file nor establishes permission to read it.
Authorize every recipient
Authentication establishes caller identity; authorization decides what that identity may access. A tenant is the customer or organizational group whose data and allowance must remain separated. Credential brokering lets a gateway supply an upstream credential after checking access, rather than giving every client the provider key. Model access and permission to disclose particular data are separate decisions.
Keep credentials narrowly scoped. Rotation replaces a credential; revocation removes its authority or usability. Short lifetimes limit exposure but do not prevent disclosure. Request text and model-generated suggestions must not expand permissions. The enforcement boundary belongs in trusted code, as explained in Independent enforcement.
Inspect effective permissions rather than the apparent strictness of one rule. Aperture grants are additive: access is the union of matching grants. A narrow grant cannot subtract permission supplied by a broad one. Its shipped configuration grants all users all models, so a restrictive policy requires removing or narrowing the grants that actually confer access.
Data eligibility attaches to the actual processing path. A Bedrock geographic inference profile can dispatch beyond the API entry Region to other Regions in its designated geography. If that destination set conflicts with approval, reject the profile rather than automatically broadening permissions. Similarly, OpenRouter documents retention at endpoint granularity; its zero-data-retention routing control excludes enabled plugins and tools. An approved inference endpoint does not approve those additional recipients.
A remote routing classifier receives information before the answer model does. A remote checker, shadow execution, and alternate provider create further disclosures. Check each recipient before sending data, and treat unknown approval as insufficient permission. Separate approval for a remote selector from approval for every possible processing destination. A permitted API entry Region does not approve the whole destination set. Approve actual service paths develops the governance decisions these checks implement.
Admission and selection
Admit bounded work
Permission to invoke a service does not establish room to do so. A rate limit bounds work over time; a quota allocates an allowance; a concurrency limit bounds simultaneous work; and a deadline sets the latest acceptable completion time. Their scopes can overlap. A tenant may have allowance remaining while its provider account has exhausted an input-token rate limit.
A token bucket is a rate-limiting mechanism whose capacity replenishes over time and can allow a bounded burst; its accounting tokens need not be language-model tokens. Claude documents separate request, input-token, and output-token limits, alongside spending allowances. Independent gateway-instance counters are not automatically one shared limit. Envoy distinguishes local limiting from coordinated global enforcement.
Admission control decides whether work may enter the bounded service. It can accept, wait within a limit, or reject. Waiting consumes the caller's time and must not become an unbounded substitute for capacity. Per-customer controls also limit a noisy neighbor's ability to exhaust shared resources. Scheduling and admission distinguishes accepting work from choosing which admitted work runs next.
Concurrent spending needs outstanding commitments, not only completed spend. LiteLLM documents reserving estimated maximum request cost before execution and replacing that reservation with priced consumption afterward. Aperture instead documents positive-balance admission followed by estimated deductions after completion. These are different contracts: the latter must not be interpreted as a strict reservation for all in-flight work.
For a small example, let a shared pool have 10 credits available and let requests A and B each require a 6-credit reservation. Independent balance checks can both see 10 and admit 12 credits of commitments. Coordinated check-and-reserve admits A, leaves 4 available, and makes B wait or fail admission. Credits are illustrative admission units.
A hard spending claim additionally depends on conservative pricing bounds, coordinated state, and defined failure handling. Unpriced work or delayed usage can weaken it into a soft limit. Retries and escalation must consume the same logical request's remaining allowance; changing the destination does not replenish the budget.
Choose among eligible routes
A route combines a model with a concrete serving path. First exclude paths that fail capability, authorization, disclosure, or resource requirements. Then apply preferences to the remaining candidates. A forbidden endpoint cannot compensate with a better latency score. If no route remains, return an explicit no-eligible-route disposition rather than quietly weakening a requirement.
A policy alias is a caller-facing name resolved through controlled configuration. It can name a logical service rather than one immutable deployment. LiteLLM groups deployments behind aliases and offers weighted, least-busy, latency-based, and rate-limit-aware selection. These operational strategies distribute work; none by itself establishes that a selected model will produce a better answer for this task.
Specify precedence rather than relying on incidental configuration order. One reasonable contract applies operator constraints first, then an explicit caller choice if allowed, then ordered task rules, then the configured placement strategy. Define tie handling where reproducibility matters. Treat health and latency observations as dated measurements: an apparently healthy endpoint can fail on the next request.
Session affinity keeps related requests on a destination, often to reuse cached state. It is a preference only while that path remains eligible. Affinity does not establish durable conversation storage or permission to retain data. These constraints leave a separate optimization question: whether cheaper or faster eligible routes meet the workload's quality targets. That comparison belongs with AI Cost and Performance Engineering.
Quality-aware decisions
Route by task evidence
A task segment is a subset of requests with meaningful shared requirements: language, output type, workflow role, difficulty, or consequence. Start with metadata the application already knows. For example, Factory's role-specific design distinguishes careful reasoning for planning, code fluency and creativity for implementation, and precise instruction following for validation. These requirements suggest what to test when assigning models; they do not establish a universal winner for each role. If evaluation supports stable assignments, explicit segment rules can make those choices without an extra predictive model.
A learned router predicts model preference from request features, or extracted properties. Its pairwise training labels identify which of two answers to the same query was preferred. That preference does not necessarily establish correctness or application success.
RouteLLM estimates from the query the probability of preferring the stronger model's answer over the cheaper alternative, before either generates. A score at least the configurable selection threshold t selects the stronger model; otherwise, the cheaper one. Raising t reduces strong-model use. Misrouting can waste resources or reduce answer quality.
Distribution fit matters. RouteLLM's February 2025 revision found that routers trained only on Chatbot Arena preferences could approach random-routing performance on MMLU and GSM8K; task-relevant training data improved routing. The revision also examined router throughput and resource costs. These findings motivate testing the selector on representative work, including unfamiliar segments, and measuring its own overhead. When evidence is insufficient, use a validated default or defer instead of interpreting a score as assurance.
Bedrock Intelligent Prompt Routing made this decision a managed interface. Its April 22, 2025 general-availability announcement described configurable model pairs and a quality-difference setting. AWS reported approximately 85 milliseconds of routing-component overhead at the 90th percentile in internal testing—not an end-to-end guarantee. Selection can reduce total latency while still adding work of its own.
Bedrock calls one quality-routing destination its fallback model even without a preceding execution failure. Classify the mechanism by its decision inputs, not the product label. Its documentation also describes English optimization and no adaptation from an application's own performance data. Evaluate the intended languages and domains, and distinguish a preference estimate from a calibrated correctness forecast.
Escalate after inspecting an answer
An acceptance gate assesses a generated candidate against requirements. An escalation cascade requests another candidate when that assessment is insufficient. It differs from pre-generation routing because an answer is now available as decision input, and from failure fallback because the first invocation may have completed normally. If no candidate qualifies, deferral explicitly declines automatic completion.
Chow's reject-option analysis, published in January 1970, formalized trading classification errors against withholding answers for exceptional handling. Rejection avoids some mistakes but also withholds answers that would have been correct. Its probability assumptions do not validate arbitrary LLM confidence scores; its enduring contribution is separating prediction from acceptance.
FrugalGPT applies response-conditioned selection to LLM calls: score the query and generated answer, stop above a stage threshold, otherwise invoke another model. Its HEADLINES experiment classified gold-price direction from financial headlines using GPT-J, J1-L, GPT-4, and a DistilBERT scorer. It reported 87.2% accuracy versus GPT-4's 85.7%, with aggregate API costs of $6.50 versus $33.10 under historical March 2023 prices. This demonstrates a workload-specific opportunity, not current prices, full operating costs, or a per-request spending ceiling.
The gate is itself fallible. False acceptance returns a candidate that violates the requirement; false rejection spends more or defers despite an adequate candidate. Valid structure is not correctness, and a stronger model need not repair a particular error. Measure coverage, the fraction accepted, alongside error among accepted cases. Raising a threshold helps only if its score usefully distinguishes failures. Deferral evaluation covers the resulting policy.
Every cascade request pays for its first invocation and applicable check. Rejected cases also pay for subsequent work, so predictably difficult requests or tight deadlines can favor direct selection. A bounded cascade must assess the second candidate too: exhausting attempts does not make the last answer acceptable. Verification and search methods are developed in Reasoning and Test-Time Compute.
Use the answer to decide whether to continue
ExampleOnce the request is admitted to this cascade, its first generation is required. Additional generation depends on rejection and remaining resources.
Read the diagram as text
- Generate candidate A. First invocation for the task.
- Assess candidate A. Evaluate the query and answer against acceptance requirements.
- Generate candidate B. Another eligible invocation for the same task.
- Assess candidate B. The last candidate still needs acceptance.
- Return accepted candidate. Acceptance is a policy decision, not infallible correctness.
- Defer automatic completion. No acceptable candidate within the permitted work.
- Generate candidate A → Assess candidate A: Candidate A and task input.
- Assess candidate A → Return accepted candidate: Accept A.
- Assess candidate A → Generate candidate B: Reject; route and resources available.
- Assess candidate A → Defer automatic completion: Reject; further work not permitted.
- Generate candidate B → Assess candidate B: Candidate B and task input.
- Assess candidate B → Return accepted candidate: Accept B.
- Assess candidate B → Defer automatic completion: Reject B.
A Unified Approach to Routing and Cascading for LLMs, by Jasper Dekoninck, Maximilian Baader, and Martin Vechev, first appeared in October 2024 and was revised in May 2025. It combines selecting the next model with deciding whether further generation is worthwhile. Its conclusions depend on estimated quality and cost; experiments using simulated quality estimates or ground-truth tests do not establish that equivalent checkers are available in deployment.
Failure and continuity
Interpret the failure before recovery
Recovery starts by identifying what failed and what is known about execution. HTTP status is only one signal. Provider codes, stop reasons, stream events, and local validation can describe different outcomes. For example, Claude documents both temporary rate limiting and exhausted monthly allowance under 429; waiting briefly does not resolve the latter.
| Observed event | What it establishes | Appropriate response |
|---|---|---|
| Invalid input or unsupported feature | The submitted request cannot be handled as specified | Correct or explicitly renegotiate; unchanged retry is not a repair |
| Authentication or authorization denial | The attempted access was not accepted | Stop and repair authority; do not bypass the restriction |
| Temporary limit or overload | The service could not accept or serve work normally | Consider bounded waiting or another eligible route |
| Exhausted spending allowance | Additional work lacks the required allowance | Stop or obtain an authorized allowance change |
| Refusal, filtering, or output limit | Generation ended with a nonordinary disposition | Preserve the reason; apply an explicit application policy |
| Malformed or semantically invalid answer | The candidate failed an output requirement | Reject or use bounded quality correction; do not label it a network fault |
| Timeout after dispatch | The caller lacks a confirmed outcome | Preserve uncertainty and assess whether repetition is safe |
| Error after output began | A partial response exists, but normal completion failed | Expose partial failure under the streaming contract |
A refusal is a decision not to supply the requested answer, not evidence that another provider should automatically be tried. Likewise, an unknown outcome after dispatch is not known non-execution. Return stable, safe categories to applications while preserving detailed provider diagnostics in protected records. Keep secrets and sensitive policy details out of public error text. Failures need distinct meanings develops the adjacent generated-output boundary.
Bound retries and alternate routes
A retry is another execution attempt. Failure fallback chooses an alternate route after an execution problem. Both consume resources, and neither is automatically safe. Coordinate attempt limits across the client, gateway, and provider SDK. Independent layers multiply attempts: three attempts at each of two layers can produce nine downstream executions. Amazon's retry guidance recommends bounded retries and backoff, increasing the delay between attempts, with randomized jitter to avoid synchronized bursts.
Propagate remaining time rather than resetting a full timeout. In the illustrative timeline, a 2,000-millisecond request spends time on admission, a failed attempt, waiting, and an alternate attempt. When the alternate fails at 1,950 milliseconds, only 50 milliseconds remain. A policy requiring at least 100 milliseconds for another attempt must stop. Deadline propagation is an application obligation when the transport or adapter does not supply it automatically.
Recovery shares one deadline
Example timingsWaiting and failed work reduce the time available to every later attempt.
Read the diagram as text
- Logical request deadline. One overall allowance; it does not reset on a route change. 0 to 2,000 ms; duration 2,000 ms.
- Admission and selection. Initial checks consume elapsed time. 0 to 150 ms; duration 150 ms. Parent: Logical request deadline.
- Attempt A. Fails without committed output. 150 to 850 ms; duration 700 ms. Parent: Logical request deadline.
- Backoff. Waiting before permitted recovery. 850 to 1,100 ms; duration 250 ms. Parent: Logical request deadline.
- Alternate attempt B. Also fails without committed output. 1,100 to 1,950 ms; duration 850 ms. Parent: Logical request deadline.
- Unused allowance; stop at 1,950 ms. Stop when attempt B fails at 1,950 ms. Leave the last 50 ms unused because another attempt requires at least 100 ms. 1,950 to 2,000 ms; duration 50 ms. Parent: Logical request deadline.
Honor provider retry guidance, including Retry-After where supplied, but never wait beyond the request's remaining allowance. A circuit breaker temporarily withholds traffic from a failing dependency. Closed permits calls; repeated failures open the circuit and fail new calls quickly; after a waiting interval, half-open permits limited probes. Successful probes restore traffic, while failure reopens the circuit. A breaker assesses dependency health, not answer quality, and does not repair the dependency.
Alternate capacity must survive simultaneous activation. Amazon describes a shipping-speed system around 2001 whose caches protected a database. When caches failed together, fallback traffic overwhelmed that database and affected other work. The lesson for model services is to examine shared dependencies and failover headroom: different endpoint names do not establish independent quota, infrastructure, or spare capacity.
Before every alternate attempt, recheck current capability, permission, disclosure policy, output state, and remaining resources. Switching models changes behavioral assumptions and requires an approved quality or degradation policy. If the gateway fails, a direct-provider bypass must not silently discard its controls. Safe repetition requires a contract, especially when the surrounding workflow can create external effects.
Respect committed output and state
Response commitment is the point beyond which replacement is no longer invisible under the chosen interface. Before consumer-visible commitment, a buffered candidate may be discarded and replaced if the rest of the contract permits it. After a prefix has been exposed, appending another model's independent answer does not generally produce a valid continuation. Report partial failure and use an explicit restart contract.
Protocol boundaries differ. gRPC commits transparent retry handling when response headers arrive; that is not a universal first-token rule. OpenRouter uses server-sent events, an HTTP event stream, to report failures after output begins while HTTP status remains 200. An idle timeout measures inactivity, whereas a maximum stream duration bounds total lifetime. Envoy documents these separately, so an active stream can avoid an idle timeout while still exceeding the application's overall deadline.
Buffering for a full-answer acceptance gate delays first visible output but permits rejection before exposure. Early delivery improves responsiveness while moving some failures into the consumer-visible state. Choose that tradeoff explicitly. Compare the same provider prefix and interruption under buffered and early delivery: the consumer has received content only in the second case.
Delivery changes what recovery must preserve
Hold the provider prefix P and interruption fixed. Only the gateway’s delivery policy changes. Events run downward; spacing does not measure time.
Buffered delivery
Consumer: No content received
Replacing the internal candidate depends on the interface, remaining resources, and other obligations.
Early delivery
Consumer: P remains observed
Preserve the exposed prefix and report partial failure. Any new answer requires an explicit restart.
The arrows carry output or notifications, not model continuation. Partial failure is neither normal completion nor proof that processing or billing stopped.
Later turns introduce a different continuity problem. Portable message history can support reconstructing the next call, but provider-held sessions and file handles need a separate transfer contract. Performance affinity is different again: Workers AI's prompt-caching documentation routes related requests toward reusable prefix state while the application still supplies the conversation. Affinity does not make that state durable or portable.
Preserve tool-call identities through interruption and restart. A proposed call is not proof of an executed effect, and cancellation does not undo a completed operation. Nor does a disconnected consumer establish that provider processing or billing stopped; cancellation support varies by provider path. Reconcile unknown tool effects before resubmitting work that could repeat them.
Accounting and explanation
Account for all attempted work
Resource accounting follows executions rather than visible answers. Connect the logical request to every metered selector, checker, candidate generation, retry, and shadow call. Preserve tenant, destination, quantity, unit, outcome, and the applicable price reference. Rejected answers can still cost money, and an interrupted attempt can leave unresolved consumption. Cost-ledger mechanics explain the underlying attribution model.
| Execution | Purpose and outcome | Reservation | Usage evidence |
|---|---|---|---|
| R1/S1 | Remote selector completed | Estimated before dispatch | Reported quantity and unit retained |
| R1/A1 | First answer attempt interrupted | Estimated before dispatch | Final usage unavailable; unresolved |
| R1/A2 | Alternate answer returned | Separate admission decision | Reported quantity and unit retained |
For streamed Chat Completions, OpenAI documents that an interrupted stream may never deliver its final usage chunk. Missing usage therefore means unknown consumption, not zero. Retain the attempt and its uncertainty until suitable evidence resolves it. A gateway must define how outstanding reservations are handled during that interval; there is no universal settlement rule that follows merely from the connection closing.
Provider reports are a separate reconciliation stream. Anthropic documents usage and cost endpoints with different coverage and delayed reporting; neither supplies the application's task-to-attempt mapping. Keep provider charges, internal quota deductions, and customer billing distinct. Deduplicating a ledger event prevents duplicate entries, not duplicate upstream execution. Broader decisions about cost per completed or useful task belong in workload economics.
Record why the route ran
A trace connects operations belonging to an execution. For routing, the essential connection is between the policy decision, the provider attempt, and its outcome. The requested alias alone cannot identify what ran. Record selected endpoints, requested and returned model identities when available, adapter versions, and the reason for each escalation or fallback. Trace mechanics and system identity provide the broader framework.
| Record | What it explains |
|---|---|
| Decision ID, policy revision, trusted classification | Which rules and inputs supported the decision |
| Eligibility exclusions and selection reason | Why a candidate was rejected or selected |
| Attempt ID, endpoint, requested and returned model | Which execution the outcome belongs to |
| Health/quality signal timestamps | How fresh the selection evidence was |
| Queue, selection, first-visible-output, and terminal events | Where elapsed time accumulated |
| Usage-record link and evidence status | What consumption is known or unresolved |
Open Policy Agent (OPA) evaluates requests against policy. Its decision logs connect policy inputs, results, revisions, and trace identifiers. To establish that an allowed request actually ran, join that decision to the gateway's dispatch and provider outcome records. OpenTelemetry's telemetry conventions likewise distinguish identifying information from proof: a provider label can reflect a proxy or compatible API rather than the actual model provider.
Decision records need not retain full prompts. Exclude credentials, minimize sensitive fields, and use restricted payload references when investigation needs deeper evidence. OPA supports masking and can also drop logs through filtering or rate limits, so enabled logging does not prove completeness. Monitor missing records and preserve the retention and access policy for the evidence itself.
Evaluation and controlled change
Evaluate the complete policy
Evaluate the policy that will actually run: eligibility checks, selection, acceptance gates, attempt limits, and recovery. A baseline is the credible alternative for the same workload. Compare a fixed eligible route and simple segment rules before crediting learned routing or cascades with improvement. Use matched tasks, preserve consequential segments, and include selector and checker work.
| Test family | Concrete requirement |
|---|---|
| Policy invariants | A forbidden recipient stays forbidden through retries, escalation, and fallback |
| Adapter contracts | Required behavior is preserved or explicitly rejected, including ignored-field cases |
| Failure fixtures | Injected timeouts, overload, and partial responses produce bounded attempts and correct terminal outcomes |
| Concurrent admission | Outstanding reservations participate in shared-limit checks; uncertain usage is not settled as zero |
| Task-quality comparison | Accepted-result quality, deferral, deadlines, and total work meet the intended workload requirements |
RouterBench compares routing on shared task outputs and costs. Its non-query-specific Zero baseline mixes models using aggregate cost–quality information. Learned routers beat it on some task collections but underperformed on ARC-Challenge and MBPP. Its cascade experiments used known answer scores with simulated scoring errors, not a deployed checker. Replacing that simulated component can change the result materially.
Sweep routing thresholds and compare the resulting outcomes, not just the fraction of strong-model calls. For cascades, report accepted-result quality together with coverage, deferral, and all attempted work. Lower average expense cannot compensate for a prohibited disclosure or an unacceptable critical-segment regression. A successful fallback fixture is narrower still: LiteLLM documents ways to trigger fallback in direct Router tests, but reaching the alternate does not establish preserved authority, deadlines, or accounting.
Selective labels arise when existing routing decisions determine which outcomes are observed. The chosen model's answer does not reveal what an uncalled alternative would have produced. Representative, authorized comparison runs can fill some gaps; ordinary production logs cannot. Incomplete feedback explains why unknown alternatives must remain unknown rather than inferred successes.
Keep policy development separate from independent assessment. Repeatedly selecting the best threshold on one finite dataset can select favorable noise. Cawley and Talbot's model-selection analysis demonstrates that a development criterion can improve while independent performance deteriorates. More elaborate routing makes independent validation more necessary, not less.
Release policy without restoring old authority
The control plane publishes routes, capability declarations, and policy; the data plane handles requests using that configuration. Publishing a version does not mean every serving instance is using it. Envoy's xDS configuration-distribution protocol, for example, distinguishes acknowledgment of valid resources from successful runtime application. Verify the applied configuration, and publish dependencies in order: introduce a destination before switching routes to it, then retire the old destination.
A canary exposes a bounded live population to a candidate policy. Rollback restores a previous operating configuration when its assumptions remain valid. Runtime routing flags can move cohorts or select a stable alternative without an emergency code release. Record who changed the policy, the affected population, observed behavior, and the version actually serving requests; choose thresholds for the workload rather than adopting generic defaults.
Availability and authorization freshness are separate. OPA can restore a persisted policy bundle when its server is unavailable, while remote distribution is eventually consistent. That preserves operation with older policy, not proof of current permission. Define acceptable staleness and stop affected work when authority cannot be established. Rollback must reapply current revocations rather than restoring every grant bundled with an older release.
Distinguish decision-only shadowing, which records the route a candidate rule would select, from execution mirroring, which sends a second request. Envoy's mirroring example delivers the input to both primary and mirror services. Discarding the mirror response does not remove that disclosure or processing cost. Authorize the additional recipient and work before using live experiment designs.
| Change | Required check |
|---|---|
| Model alias or adapter changes | Resolve actual identity; rerun capability and task-quality checks |
| Destination or retention approval changes | Recompute every affected recipient path, including recovery and mirroring |
| Task mix changes | Reassess segment outcomes, selector fit, and acceptance thresholds |
| Dependencies fail together | Test alternate-path headroom and coordinated attempt limits |
| Policy distribution fails | Check applied versions and the permitted freshness window |
Keep a fixed route when it meets the requirements and additional selection does not earn its cost. Add segment rules when measured differences are stable and understandable. Introduce learned routing or cascades when their complete-policy results justify prediction, checking, and operational complexity. Remove them when that benefit no longer holds. Dependable routing preserves obligations first and optimizes within them.
Open questions
Reliable routing under changing workloads remains difficult because model behavior, task mix, and observed labels change together. Progress would mean maintaining segment-level benefits against simple baselines while accounting for unobserved alternatives and the cost of collecting new comparisons.
Production acceptance gates remain a limiting component of cascades. Benchmark access to known answer scores can hide the difficulty of recognizing a good answer at runtime. Progress would mean validated checkers that retain useful coverage under shift without consuming the savings or accepting consequential errors.
Interrupted-request accounting needs explicit reconciliation contracts. Missing final usage and delayed provider reports make immediate exact settlement difficult. Progress would mean tested handling of concurrent reservations, late reports, repeated callbacks, and unresolved consumption without either double charging or inventing zero usage.
Cross-provider continuity remains constrained by exposed output and provider-owned state. Progress would require explicit, testable restart and state-transfer interfaces that preserve tool identities, permissions, and already observed effects—not merely a common request shape.
Fast revocation and resilient policy distribution can conflict during outages. Progress would mean observable, workload-appropriate freshness bounds and recovery procedures that restore service without resurrecting withdrawn authority.
