Distributed Training and Inference

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. vLLM 0.22.0: Parallelism and Scaling

   vLLM supports tensor- and pipeline-parallel inference, including one executable model spread across multiple nodes. Its example combines tensor parallelism of eight with pipeline parallelism of two on two eight-GPU nodes. After weights fit, available KV-cache capacity still constrains concurrent request state. The documented maximum-concurrency log is an estimate based on configured maximum request length. Multi-node execution requires consistent model paths and software environments.

2. vLLM: Data Parallel Deployment

   vLLM replicates model weights across data-parallel engines to process different request batches, with an independent KV cache per engine. An engine can itself own multiple tensor-parallel GPU workers. However, MoE configurations combining data-parallel attention with shared tensor- or expert-parallel expert layers couple those engines: forward passes must align, and idle ranks may execute dummy forwards while other ranks have requests. Separate dense-model servers can instead operate independently behind an external request balancer.

3. LLNL: Introduction to Parallel Computing Tutorial

   Distributed-memory processors own local memory and communicate to obtain remote data; aggregate memory is not automatically one shared address space. Communication latency describes minimal-message delay, while bandwidth measures transferred data per time. Many small messages can become latency-dominated, and competing traffic can saturate a network. Strong scaling holds total work fixed while increasing processors; weak scaling holds work per processor fixed. Communication, synchronization, and insufficient work granularity can outweigh parallel computation. At a barrier, the slowest participating task determines when others can proceed.

4. LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication

   The study held inference workloads fixed while increasing GPUs. For Llama 3.1 70B, vLLM tensor-parallel latency improved from four to eight GPUs but then generally flattened or increased. YALIS profiling from eight to sixteen GPUs showed communication increasing about 1.6× in a decode-heavy case, offsetting reduced computation. Its BF16 decode all-reduce example with batch eight and hidden width 8192 carries 128 KB. Hybrid tensor/pipeline execution reduced communication but did not necessarily reduce small decode matrix-operation time.

5. PyTorch: Training a Classifier

   The CIFAR-10 example converts RGB images into normalized tensors and batches them as B×3×32×32, paired with B integer class labels. Learned convolution and linear weights transform inputs through convolution, ReLU, pooling, and flattening into B×10 logits. Cross-entropy compares logits with target classes; for one example it is -log softmax(logits)[target]. Training clears old gradients, runs the forward pass, calls loss.backward() to accumulate parameter derivatives by the chain rule, then optimizer.step() updates parameters using SGD with momentum. Fixed-parameter inference retains preprocessing and the forward pass, while target labels, training loss, backpropagation, and optimizer updates are unnecessary. The example uses no_grad() during testing.

6. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

   The illustrated inference loop has a prefill stage for the prompt followed by incremental decoding, with different computation shapes.

7. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

   Retaining key and value representations avoids repeating earlier prompt work during decoding, at the cost of GPU memory.

8. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

   ZeRO addressed redundant training-state storage while retaining relatively large local computations. Its mixed-precision Adam accounting uses 16 bytes per parameter: FP16 parameters and gradients plus FP32 master parameters and two optimizer arrays. Activations and temporary buffers are additional. Its communication analysis replaces gradient all-reduce with reduce-scatter and updated-parameter all-gather without increasing modeled volume for optimizer-plus-gradient sharding. Adding parameter sharding requires parameter collection for both forward and backward computation, increasing modeled volume from 2Ψ to 3Ψ elements per step.

9. vLLM memory profiling: allocations and transient peaks

   vLLM distinguishes memory outside its instance, PyTorch-controlled allocations inside it, and its allocations outside PyTorch. The profiling example separates model weights, transient activation tensors, NCCL memory and attention-backend buffers. Snapshots record current and peak framework allocations and device free memory; reserved framework memory differs from live allocated tensors. Non-KV demand includes persistent consumption and headroom for transient peaks, rather than weight bytes alone. Workspace and communication buffers can therefore reduce cache capacity even when the weights fit.

10. Transformers: cache allocation versus retained history

    KV caches retain attention keys and values beyond model weights. DynamicCache grows as tokens arrive; StaticCache preallocates a maximum capacity, masking unused positions and potentially wasting memory and attention work on short sequences. Sliding-window or chunked layers stop growing at their retained-history limits even when the configured sequence capacity is larger. Offloading transfers most layer caches through CPU memory; quantized caches reduce precision but can increase latency for short contexts. Operational inference: allocated slots are a storage decision, not evidence that the model supports that many sequence positions. For ordinary full-history static caching, reserve sufficient slots for prompt plus intended output while independently enforcing the model's supported context limit. Windowed cache capacity must instead follow the architecture's retention rules.

11. vLLM GPU worker: profiling available KV capacity

    The GPU worker profiles a model forward pass and, when configured, estimates CUDA-graph memory separately. Its available KV-cache budget subtracts profiled non-KV consumption and the applied graph estimate from requested memory. Later warmup compares actual graph-pool use with its estimate and reserves additional headroom in suggested allocations. This connects weights, activations, runtime buffers and graph storage to the space left for request state. An explicit cache-byte override skips normal capacity inference and requires a suitable value.

12. NCCL: Creating a Communicator

    A communicator associates participating CUDA devices with unique ranks from zero through n−1. Reusing one device as multiple ranks in the same communicator is unsupported. Some network errors arrive asynchronously and leave operations unable to complete. NCCL documents polling asynchronous error status, aborting failed communicators, and recreating communication state. Its error table distinguishes invalid arguments that leave the communicator usable from errors fatal to the communicator.

13. PyTorch: torchrun Elastic Launch

    Under torchrun's documented worker-failure policy, a worker failure stops and restarts all workers up to the configured restart limit. Node arrivals and departures also stop existing workers and form a new group with new rank and world-size assignments. Surviving workers are terminated during these events, so progress must be checkpointed. Node and agent failures additionally depend on job-manager policy.

14. PyTorch: Getting Started with DeviceMesh

    DeviceMesh organizes devices into named dimensions and manages the corresponding process groups. The tutorial distinguishes shard groups from replica groups and illustrates a two-dimensional replicate-by-shard arrangement. Its hybrid sharding example performs FSDP within a host and data-parallel replication across hosts. Submeshes can select separate communication groups for composed parallelism while reusing communicators.

15. PyTorch: Getting Started with Fully Sharded Data Parallel (FSDP2)

    FSDP2 shards parameters outside computation, all-gathers them before forward and backward use, reduce-scatters local gradients, and updates parameter shards with sharded optimizer state. Applying fully_shard to individual layers allows other layers to remain sharded while one executes. Prefetching can overlap the next layer's all-gather with current computation. Prefetching multiple layers increases resident memory; delayed CPU issuance can prevent overlap, and the first gather can remain exposed. Persistent shard size therefore differs from execution-time residency.

16. Building Generative Image & Video Models at Scale

    Batch-level data parallelism eventually needs to be supplemented by model parallelism across chips.

17. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Partitioning a linear layer's weight matrix by output columns produces independent output pieces, allowing an elementwise nonlinearity to run locally. Partitioning the contracted dimension instead produces partial sums that must be combined before a nonlinearity. Megatron pairs a column-partitioned first MLP matrix with a row-partitioned second matrix, avoiding communication between them and reducing the second matrix's partial outputs. Its MLP uses one forward all-reduce and one backward all-reduce. The original transformer block uses two of each. Activation recomputation saves retained intermediates by repeating forward work during backward.

18. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    The speaker places tensor parallelism typically within a node and describes pipeline parallelism as sequential handoff across nodes.

19. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

    Yanping Huang and colleagues at Google introduced GPipe in a November 2018 preprint, followed by the 2019 NeurIPS paper. It partitions sequential layers using estimated computational costs and overlaps microbatches across stages. Gradients accumulate before a minibatch-level parameter update. Rematerialization retains boundary activations and recomputes internal intermediates during backward execution. The published demonstrations include a 557-million-parameter image model and a six-billion-parameter multilingual translation model, establishing capacity beyond one accelerator rather than merely proposing a schedule.

20. NCCL: Collective Operations

    All-reduce combines corresponding array elements across ranks and returns the reduced array to every participant. All-gather instead concatenates each rank's contribution in rank order. Reduce-scatter reduces corresponding values, then distributes equal-sized portions of the result among ranks. Broadcast copies a designated root rank's buffer to all participants. Reduce-scatter followed by all-gather implements the result of all-reduce. NCCL requires matching counts and datatypes across participating ranks; violations can hang, crash, or corrupt data.

21. MPI 4.1: Nonblocking Collective Operations

    A nonblocking collective initiates communication and returns a request handle; a separate completion operation establishes when its local buffers are safe to access or modify. Initiation alone does not establish completion. Local completion does not generally establish that other processes have completed. All participants must initiate collectives in the same order on a communicator. Nonblocking operations permit computation overlap, but that permission is not a measured performance guarantee.

22. SUMMA: Scalable Universal Matrix Multiplication Algorithm

    Robert van de Geijn and Jerrell Watts describe distributed matrix multiplication on a logical two-dimensional processor grid. Workers retain their output blocks while receiving the required input panels through row and column broadcasts. Blocking and pipelined broadcasts reduce the cost of repeatedly initiating small transfers. The paper models communication as message startup plus a size-dependent transfer cost; equivalently, T_message ≈ α + S/B under the model's assumptions. This explains why dividing work more finely can reduce local arithmetic while increasing communication overhead.

23. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    Overlap can hide part of communication time, but the speaker reports that substantial network waiting remained in customer workloads.

24. Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities

    Gene Amdahl's 1967 IBM paper challenged the assumption that adding processors would automatically accelerate complete applications. It emphasized sequential housekeeping, irregular computation, data management, and memory contention alongside parallel arithmetic. Improving only the readily parallelized portion leaves these other costs exposed. The contribution is a whole-workload argument: parallel hardware must improve the actual problem, including its coordination and data movement, rather than only a favorable computational fragment.

25. Reevaluating Amdahl's Law

    John Gustafson's 1988 Sandia paper asked what happens when additional processors enable a larger problem within approximately the same execution time. If s is the serial fraction measured on the parallel machine and N is the processor count, its scaled-speedup model is s + N(1-s). This differs from measuring how quickly more processors finish one fixed problem. The paper connects the argument to scientific simulations whose parallel work grows with problem size while serial overhead changes relatively little.

26. NCCL Tests: Performance

    NCCL Tests distinguishes algorithm bandwidth, the operation's data size divided by elapsed time, from a calculated bus-bandwidth normalization. For all-reduce over n ranks, its correction multiplies algorithm bandwidth by 2(n−1)/n; all-gather and reduce-scatter use (n−1)/n. These factors account for the communication work associated with the collective rather than changing the measured duration. The documentation separately identifies startup-dominated small messages and bandwidth-dominated large messages.

27. PyTorch: Distributed Data Parallel

    DDP initializes replicas from a shared model state and groups parameter gradients into communication buckets. Backward hooks mark gradients ready; once a bucket is ready, its reducer launches asynchronous all-reduce to obtain mean gradients. Backward waits for reductions before exposing synchronized gradients to the optimizer. Identical starting state and averaged gradients allow each replica's local optimizer to maintain matching parameters. Reductions execute in bucket order across processes, not arbitrary readiness order. Poor alignment between bucket contents and gradient readiness delays communication launch.

28. DeepSpeed: Zero Redundancy Optimizer

    ZeRO progressively partitions training state: stage 1 partitions optimizer state, stage 2 also partitions reduced gradients, and stage 3 also partitions parameters, gathering and repartitioning them during execution. Each process updates its owned optimizer partition. The tutorial's 1.5-billion-parameter GPT-2 example reports Adam-related state decreasing from 18 GB per device to 2.25 GB across eight data-parallel ranks, enabling a configuration that otherwise runs out of memory.

29. Accelerate: Performing Gradient Accumulation

    For variable-length token tasks, the intended token-mean loss uses summed losses divided by the total number of scored, non-padding tokens across the accumulation window. Averaging per-batch means is different. The distributed example gathers the token count across processes, compensates for DDP's gradient averaging and Accelerate's accumulation scaling, and suppresses intermediate synchronization. Its published teaching fixture compares an accumulated update with a cloned model updated on the combined batch.

30. PyTorch 2.9: DistributedSampler

    DistributedSampler assigns dataset subsets using replica count and rank. When dataset length is not evenly divisible, drop_last removes the tail; otherwise the sampler adds indices to equalize partitions. Processes must use the same shuffle seed. Calling set_epoch before constructing each epoch's iterator changes the shuffle order; otherwise the same ordering repeats. The documented sampler assumes a constant-size dataset with stable element ordering.

31. Horovod: Fast and Easy Distributed Deep Learning in TensorFlow

    Alexander Sergeev and Mike Del Balso's February 2018 Uber paper addressed both distributed-training performance and the integration burden of TensorFlow's parameter-server approach. Horovod built on Baidu's ring-all-reduce implementation, subsequently using NCCL, and exposed gradient averaging through an optimizer wrapper with initial parameter broadcast. Cross-worker timelines revealed many small reductions in deep models. Tensor Fusion packs ready tensors of matching datatype into one buffer, reduces that buffer, and copies results back. The reported Inception V3 and ResNet-101 experiments showed improved scaling over the tested distributed TensorFlow baseline.

32. A Bridging Model for Parallel Computation

    Valiant's August 1990 paper proposed bulk-synchronous parallel computation as an intermediate model connecting portable software with parallel hardware. A superstep combines local computation and message exchange; participants advance after completion is established for the participating components. The model separates processing, communication, and synchronization rather than assuming that additional processors alone determine execution time.

33. Large Scale Distributed Deep Networks

    Google's 2012 DistBelief system combined partitioning within a model with multiple model replicas. Downpour SGD let replicas independently fetch parameters and send gradients to a parameter server whose state was itself sharded across machines. Other replicas could continue when one replica failed, but gradients could use outdated parameters, and independently updated server shards need not share an update count or ordering. This traded stricter coordination for tolerance of variable worker progress.

34. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    Yanli Zhao and colleagues' 2023 Meta paper presents FSDP as a PyTorch-native realization of ZeRO-inspired state sharding. Its engineering contribution includes integration with the tensor dispatcher and memory allocator, initialization when the complete model cannot fit on one GPU, and control of prefetched parameter allocations. These concerns matter because reducing persistent shard size alone does not prevent initialization or transient-memory failures. The paper evaluates the system on clusters of up to 512 A100 GPUs.

35. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    Activation checkpointing reduces stored activations by recomputing them during the backward pass, but its configuration must control added computation.

36. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    Transformer-block inputs can be moved off the GPU while unused and prefetched for backpropagation to reduce GPU activation storage.

37. PyTorch: Numerical Accuracy

    Floating-point addition and multiplication are not associative, so changing operation order can change numerical results. PyTorch does not guarantee bitwise identity for mathematically identical computations, across platforms, or across releases. Batched and individually computed matrix operations can differ. Different reduction orders and accumulation precision can also change whether extreme inputs produce finite values.

38. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

    Pipeline parallelism assigns layers to devices and schedules microbatches through them. Flushes preserve consistent forward/backward weight versions. GPipe's all-forward-then-all-backward schedule retains activations for all microbatches; synchronous 1F1B bounds outstanding activations by pipeline depth but retains the same bubble time in the paper's model. Bubble time relative to ideal computation is (p−1)/m. Combined execution uses p×t×d devices and global batch b×m×d. The paper's fixed-resource experiments show that excessive tensor parallelism increases communication and shrinks local matrix operations; its cluster favored tensor groups within eight-GPU nodes and pipeline transfers across nodes.

39. PipeDream: Generalized Pipeline Parallelism for DNN Training

    PipeDream's SOSP 2019 design overlapped different minibatches to avoid repeated pipeline flushing. Weight stashing retained parameter versions so a stage's backward pass used the version from that minibatch's forward pass. This did not guarantee that different stages used one common version. Optional vertical synchronization aligned versions across stages, but updates still applied gradients computed from older weights. Its one-forward-one-backward schedule therefore did not imply the same update semantics as synchronous GPipe.

40. NVIDIA: Massively Scale Your Deep Learning Training with NCCL 2.4

    NVIDIA describes ring all-reduce as bandwidth-efficient but with latency growing linearly with participant count. Its double-tree implementation divides data across complementary trees to obtain logarithmic latency while retaining bandwidth efficiency. The hierarchical ring example performs reduce-scatter within nodes, all-reduces between nodes, then all-gather within nodes. Reported Summit experiments showed bandwidth degradation when communication crossed additional network-switch levels. NCCL selected rings when they provided greater bandwidth.

41. NVIDIA GPUDirect RDMA: Overview and Design Considerations

    GPUDirect RDMA enables direct data exchange between GPU memory and a peer device such as a network adapter through PCI Express. Direct payload movement still requires software to identify and register the memory, arrange mappings, initiate transfers, and release resources safely. The documentation distinguishes paths through PCIe switches, CPU I/O infrastructure, and inter-socket links because those paths have different performance and support constraints. GPU memory registration can itself be expensive, motivating reuse of registrations.

42. Doubling All2All Performance with NVIDIA Collective Communication Library 2.12

    NVIDIA's February 28, 2022 account explains PXN, which can move GPU data over NVLink to an intermediate GPU before sending it through that GPU's network interface controller, or NIC. This changes which physical links carry the payload. In a rail-optimized network, corresponding NICs across servers share a leaf switch; choosing an intermediate GPU on the destination's rail can avoid traversing higher-level switches. PXN also aggregates messages destined for the same remote node.

43. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    In the presented rail-optimized topology, NCCL PXN uses an internal NVSwitch hop to reach the appropriate rail without routing through the spine.

44. Kubernetes: Pod Topology Spread Constraints

    A failure domain groups resources exposed to a common failure, such as one node or zone. Kubernetes illustrates why placing both replicas on one node permits a single node failure to remove the workload. Topology labels identify domains, and spread constraints govern placement across them. Hard constraints can leave a workload pending; preferences allow scheduling despite imbalance. Placement also affects traffic between zones and its latency.

45. Kueue: All-or-nothing Scheduling

    Gang scheduling addresses workloads whose cooperating processes must run together: partially started jobs can occupy resources while waiting for missing participants. Kueue distinguishes reserving aggregate quota from checking physical placement. Eight available GPUs split across two four-GPU nodes cannot satisfy one eight-GPU Pod. Topology-aware admission checks node and domain capacity. A separate readiness timeout can evict and requeue partially ready workloads, releasing their quota.

46. Infra behind Krea 2 - How to train and serve at scale

    Krea combined Kueue gang scheduling and workload priority with Kubernetes pod priority, but manually maintained resource quotas could become stale.

47. Insights from Snorkel AI running Azure AI Infrastructure

    Treat low accelerator utilization as evidence of a pipeline bottleneck: investigate inter-node networking for multi-node jobs and data loading for single-node jobs, including shared-filesystem read throughput.

48. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    Fully sharded data parallelism can reduce model memory while leaving attention activations as the limiting allocation.

49. Megatron Core: Context Parallel Package

    Megatron distinguishes sequence parallelism that splits selected Dropout and LayerNorm activations from context parallelism that partitions inputs and activations along sequence positions. Token-local operations can operate on those partitions, while attention needs remote keys and values. The documented context-parallel implementation exchanges KV through ring point-to-point communication, gathers it again for backward computation, and reduces corresponding KV gradients. Context parallelism composes with tensor, pipeline, and data parallelism.

50. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

    Ulysses starts with each worker owning different sequence positions. Before attention, an all-to-all redistributes queries, keys, and values so each worker receives the full sequence for a different subset of attention heads. After local attention, another all-to-all restores sequence-position partitioning for subsequent operations. The design targets long-sequence memory and communication constraints and can combine with ZeRO-3 for model-state sharding.

51. Ring Attention with Blockwise Transformers for Near-Infinite Context

    Hao Liu, Matei Zaharia, and Pieter Abbeel's October 2023 paper distributes sequence blocks across devices. Each device keeps its query block while key/value blocks circulate around a ring, combining attention contributions incrementally. Communication can overlap block computation when computation lasts at least as long as the corresponding transfer. Consequently, faster arithmetic relative to the interconnect requires larger blocks to hide communication. The experiments demonstrate increased training sequence capacity using GPU and TPU configurations, combining Ring Attention with model-state sharding.

52. From Mixture of Experts to Mixture of Agents … with Super Fast Inference

    Mixture of Experts (MoE) replaces a monolithic feed-forward network with multiple expert networks and routes each token to a subset.

53. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    GShard's mixture-of-experts layer selects at most two feed-forward submodules per token and combines their outputs using routing weights. Experts reside on different devices while other layers are replicated. All-to-all changes ownership from input groups to experts: each participant sends different slices to different peers, which assemble received slices. Uneven expert selection can overload a few buffers and leave other experts underused. GShard caps expert capacity; tokens overflowing both selected experts bypass expert computation through residual connections.

54. FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models

    The FasterMoE authors describe expert shadowing as an alternative to sending every token to an overloaded expert's original device: selected popular expert weights are broadcast so other workers can perform that expert's computation. A performance model guides when replication is worthwhile. The system also divides communication and computation into smaller groups to overlap them. Its topology-aware routing proposal additionally changes expert selection, which is a model-routing intervention rather than merely a different implementation of unchanged communication.

55. From Mixture of Experts to Mixture of Agents … with Super Fast Inference

    Mixture of Agents (MoA) combines already pretrained LLMs through orchestration rather than requiring MoE pretraining.

56. NVIDIA Dynamo: KV-Aware Routing

    Dynamo's router combines reusable prefix-cache overlap with projected active prefill and decode load. A worker with matching cached state can avoid repeated prompt computation, but a colder worker can win when the cache-rich worker is busier. Workers publish cache creation and release events to update the index. Worker selection and KV transfer are separate responsibilities: the router selects destinations while backend transfer mechanisms move state.

57. Automatic prefix caching — vLLM

    A repeated shared token prefix can reuse stored KV state and skip computation for that prefix. Stable initial context favors reuse; unrelated prefixes cannot share those computations merely because their meaning is similar. The benefit is in prefill, not generating new decode tokens, so long outputs may limit the fraction of request latency saved.

58. Taking Reinforcement Learning Cross Datacenter

    Keep the tightly coupled trainer on fast fabric, but distribute rollout serving islands across regions or providers.

59. Taking Reinforcement Learning Cross Datacenter

    Publish immutable policy versions and make requested, acceptable, and returned versions explicit.

60. Hacking the Inference Pareto Frontier

    Disaggregation lets prefill and decode use different resource allocations and avoids competition between their scheduling needs.

61. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    DistServe frames serving capacity around request rate subject to both TTFT and TPOT objectives and an explicit attainment target. Its 13B-model experiment compares decode-only batches with batches containing one additional prefill job: prompt work delays continuing generation, with stronger interference for longer prompts. The paper distinguishes execution time from latency that also includes queueing. Chunking prompt work reduces long decode stalls but can add repeated KV reads and prompt-processing overhead.

62. Splitwise: Efficient Generative LLM Inference Using Phase Splitting

    Splitwise assigns prompt and decode machines together so cache transfer can begin as individual layers finish. Its implementation uses one-sided writes and a semaphore the destination waits on before proceeding. Layerwise transfer overlaps later prompt computation but introduces synchronization and interference; short prompts can favor serialized transfer. Figure 14 reports approximately 8 ms exposed transfer on its A100 setup and 5 ms on H100, using 200- and 400-Gbps InfiniBand respectively. These are remaining delays after overlap, not total transfer durations.

63. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    DistServe transfers KV state and the first token from prefill to decoding workers. Its placement search jointly considers phase-specific parallelism, instance counts, workload, memory, and network constraints. Variable prompt lengths can create pipeline imbalance. Cache-transfer demand grows with prompt length and arrival rate; constrained cross-node bandwidth changes feasible placement.

64. vLLM: NixlConnector Compatibility Matrix

    NixlConnector checks compatibility during prefill/decode handshake, including software versions, model dimensions and dtype, attention backend, cache dtype, and transfer protocol. Different tensor-parallel degrees and block sizes are supported only under stated restrictions. Cache layout determines whether head splitting is possible. For MLA, cache is replicated across tensor-parallel workers rather than divided into KV heads. Runtime-generated quantization scales are unsupported because those scales are not transferred with the cache.

65. vLLM 0.15.0: NixlConnector Usage Guide

    The guide distinguishes failing a request after KV-load failure from recomputing failed blocks on the decode instance. Recomputing can interfere with ongoing decoding by introducing prompt work. If an aborted request's decoder has not read the producer's cache, the producer releases those blocks after a configurable timeout, documented as 480 seconds by default. Cancellation can therefore leave source state retained temporarily.

66. Hacking the Inference Pareto Frontier

    Disaggregation is workload-sensitive: short inputs reduce its scheduling benefit, extreme operating points may favor aggregation, and a poor worker ratio can create idle capacity or queues.

67. Frontier AI at Home (literally)

    KV-cache transfer must be fast enough to overlap computation; otherwise communication adds a serial delay before decode.

68. Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts

    A PyTorch NCCL-watchdog timeout can result from CPU divergence, GPU hangs, incompatible collective arguments, or network and hardware problems. The reporting rank and timed-out operation need not originate the fault. Flight Recorder records collective type, sequence ID, tensor shapes and dtypes, call stacks, and scheduled/started/completed states. Diagnosis aligns records within each process group across ranks. Similar state mismatches can have different causes, requiring hardware signals and repeated-failure patterns.

69. Infra behind Krea 2 - How to train and serve at scale

    Collect communication-fabric telemetry beyond basic GPU metrics, covering both cross-node InfiniBand and intra-node NVLink failures.

70. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

    MegaScale monitors executor heartbeats, process status, logs, and network traffic. On detected failure, its driver suspends training across executors, runs diagnostics, replaces identified faulty nodes, and resumes from a checkpoint. Checkpointing first stages GPU state in host memory, then writes it asynchronously to distributed storage. Recovery reduces storage contention by having one worker read state shared by a group and broadcast it. The report also describes stragglers that appeared normal in isolated matrix benchmarks but slowed distributed execution.

71. TorchSnapshot: Getting Started

    TorchSnapshot requires the application to identify stateful objects to capture, including models, optimizers, learning-rate schedulers, and custom progress state. Capturing its RNGState utility restores the supported global random-generator state. Distributed snapshots involve all ranks. Its experimental world-size changes support replicated objects and supported sharded tensors; other objects can only be restored by their saving rank. Tensor resharding therefore does not automatically make arbitrary application state elastic.

72. PyTorch: Asynchronous Saving with Distributed Checkpoint

    Asynchronous checkpointing separates staging state into CPU memory from writing it to storage. The fully asynchronous example exposes distinct staging-completion and upload-completion futures. It waits for staging before the optimizer modifies parameters, allowing the copy to overlap intervening forward/backward computation without racing the next update. It waits for upload completion before queuing another checkpoint and before final cleanup. Background staging consumes CPU resources and additional memory.

73. PyTorch: Distributed Checkpoint

    Distributed Checkpoint saves and loads from multiple ranks, producing multiple files and supporting load-time resharding into another cluster topology. Loading writes into storage allocated by the destination model. Its StorageWriter contract separates local planning, coordinator global planning, per-rank data writes, and coordinator finish. Finish receives write results from all ranks, writes metadata, and marks the checkpoint successful.

74. PyTorch DCP utils.py: coordinated exception handling

    DCP's distributed all_reduce wrapper runs a local function on each rank and gathers either its result or a wrapped exception. The coordinator invokes the finishing function only when the gathered results contain no failures. Otherwise it broadcasts a CheckpointException, which participating ranks raise. Exceptions from the coordinator's finishing function are also propagated.

75. PyTorch DCP filesystem.py: FileSystemWriter and FileSystemReader

    The local filesystem writer defaults to sync_files=True: data files are flushed and fsynced before their write results are returned. finish builds storage metadata from returned results, serializes .metadata.tmp, flushes and fsyncs it, then renames it to .metadata. The reader opens the final metadata filename, not the temporary file; a fresh interrupted save without final metadata therefore fails at metadata loading. finish does not independently reread shards or compare every planned item with returned results. Existing metadata is deleted before replacement, and the inspected path contains no parent-directory fsync.

76. TorchData: Stateful DataLoader

    StatefulDataLoader supports mid-epoch state capture and restoration. Its default records yielded-batch progress and fast-forwards the sampler or dataset; custom state methods allow those components to preserve more precise state. It aggregates state across local data-loading subprocesses, explicitly not across distributed ranks. The documentation also warns that out-of-order batch delivery currently has no state-management guarantees.

77. A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps

    John Daly's Los Alamos analysis separates useful computation, checkpoint writing, work repeated after failure, and restart time. Saving more often reduces expected lost work but consumes more execution time. It revisits the first-order useful-computation interval approximation sqrt(2δM), where δ is checkpoint duration and M is mean time between interruptions, then develops a higher-order model incorporating additional failure and restart effects. The decision is therefore a balance of measured saving cost and failure exposure, not simply a preference for frequent checkpoints.

78. NVIDIA Dynamo 0.9.1: Fault Tolerance

    Dynamo distinguishes request recovery, worker health, overload handling, and infrastructure availability. Planned shutdown invalidates endpoints, stops new admission, optionally drains in-flight work, and cleans up resources. Unexpected worker loss is detected through service-discovery lease expiration; new requests then route to remaining workers. Cancellation propagates through request chains. The documentation identifies worker-failure migration tests and GPU/network fault injection as separate validation paths.

79. NVIDIA Dynamo: Overall Architecture

    Dynamo separates worker health detection and discovery-based endpoint removal from Kubernetes workload management. The Dynamo Operator reconciles DynamoGraphDeployment resources. Grove-backed multinode deployments represent worker groups through PodCliqueSet and PodClique resources, with PodCliqueScalingGroup representing coordinated scaling. Request migration and cancellation handle in-flight requests separately from these infrastructure responsibilities.

80. NVIDIA Dynamo: Health Check Reference

    A healthy frontend can return HTTP 200 even when its discovered worker list is empty. Worker health endpoints report notready before readiness and ready afterward. Optional active canaries send a backend-specific minimal inference request through the normal endpoint, exercising request handling and engine execution rather than only process reachability.

81. Dynamo Operator: worker container defaults

    The inspected operator code configures separate startup, liveness and readiness probes for workers. Its comments explicitly state that Kubernetes worker readiness does not itself determine traffic eligibility: worker registration uses an external store and the transport does not use a Kubernetes Service. The default environment in this implementation sets active Dynamo health checks to false.

82. NVIDIA Dynamo: Request Migration

    Dynamo can retain prompt and generated-token state for replay on another worker after failure. Migration is disabled by default and bounded by a configured attempt limit; exceeding a configured total-sequence-length limit disables tracking and migration. Multi-choice requests are unsupported because the migration path retains only one continuation state. Structured-output requests are also unsupported: replaying prior tokens as context does not advance a newly initialized constraint state machine, which can corrupt continuation. Migration duration metrics distinguish success, failure, and cancellation.

83. Dynamo migration.rs: response tracking and stream recreation

    The inspected migrator tracks a response before returning it upstream: it appends returned token IDs to the retained request and subtracts their count from the remaining generation budget. On a migratable error it replaces the backend stream using that accumulated request rather than emitting the retained prefix again. Each new attempt keeps the request ID, links its context to the parent and checks whether the parent is stopped or killed before dispatch. Thus its replay boundary is tokens observed by the migrator, not a client acknowledgement cursor.

84. NVIDIA Dynamo: Request Cancellation Architecture

    Frontend client disconnection cancels the request context, and linked child contexts propagate that cancellation downstream. Workers receive cancellation through control messages or connection loss, but their engine implementations must observe the context and stop computation. The runtime cancellation metric counts received signals, not confirmed engine aborts. Graceful stop does not invalidate results already in the stream; kill expresses a preference to avoid draining, with engine-specific support.

85. CUDA C++ Best Practices Guide

    Kernel launches return before GPU work completes; CPU timing must account for completion. CUDA events record device timestamps when reached in a stream, and the example waits for the stop event before reading elapsed time. Extra synchronization can alter execution. Bandwidth is data transferred per unit time. Occupancy is active warps per multiprocessor divided by its supported maximum; registers, shared memory, and block size constrain residency, and higher occupancy need not improve performance. Amdahl's law bounds fixed-workload improvement by the fraction accelerated: unchanged work remains even if the optimized portion becomes arbitrarily fast.

86. MLCommons: MLPerf Training Benchmark

    MLPerf Training measures wall-clock time to reach a specified quality target on a specified dataset. It repeats runs and aggregates results to account for variability. The Closed division constrains model choice to support comparable system measurements, while the Open division allows broader changes.

87. AIPerf Metrics Reference

    AIPerf measures TTFT from request start to the first nonempty content response. Its metric named ITL is a per-request average: (request latency minus TTFT)/(output sequence length minus one), requiring at least two output tokens. It separately measures inter-chunk arrival gaps. The displayed request-latency formula ends at the last content response, so it does not independently establish a later terminal-event timestamp. Goodput counts requests satisfying configured SLOs per benchmark second; successful requests and errors have separate counters.

88. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    Validate network benchmark gains against time to train a particular model rather than treating microbenchmark improvement as the final outcome.

89. Compute & System Design for Next Generation Frontier Models

    The speaker reports a ByteDance case in which removing a slow but nominally functioning GPU improved training throughput, illustrating the impact of stragglers in synchronous training.

90. Frontier AI at Home (literally)

    The speaker reports that low-latency RDMA made fine-grained parallel inference practical across Macs by reducing repeated synchronization costs.

91. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    Loss and MLP computations can require large sequence-length buffers even after attention memory is reduced.

92. Hacking the Inference Pareto Frontier

    Route using both reusable prefix state and current worker load; maximizing KV match alone can increase queueing.

93. MPI: A Message-Passing Interface Standard, Version 1.1

    MPI sought a portable, efficient interface for message-passing programs, giving application developers a common contract and vendors routines they could optimize. Its introduction describes collaboration among vendors, universities, laboratories, and industry, building on existing communication systems. The document identifies MPI 1.0 with June 1994 and MPI 1.1 with June 1995; the latter clarified and corrected the earlier specification.

94. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    NVIDIA's Mohammad Shoeybi and colleagues first submitted Megatron-LM in September 2019. The inspected March 2020 revision reports converged 8.3-billion-parameter models on 512 GPUs; its scaling-efficiency experiment enlarges the model rather than holding total work fixed.