Transformers and Attention

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code

   The demonstrated attention head distributes attention over available earlier positions, with future positions excluded; some patterns are interpretable, but not every head has an obvious linguistic explanation.

2. PyTorch MultiheadAttention

   A key-padding mask excludes designated source positions from attention, whereas an attention mask restricts particular query-to-source interactions. PyTorch exposes these separately: key_padding_mask identifies ignored source entries for each batch item, while attn_mask describes permitted or forbidden position pairs. When both Boolean masks are supplied, their exclusions are combined. Excluding padded source positions is distinct from enforcing causal order.

3. nanoGPT train.py

   The batch builder takes input tokens from data[i:i+block_size] and targets from data[i+1:i+1+block_size]. Each input position therefore aligns with the next recorded token. Constructed example: a stored sequence [A,B,C,D] supplies inputs [A,B,C] and targets [B,C,D]. All preceding input tokens come from the recorded sequence rather than sampled predictions.

4. RoFormer: Enhanced Transformer with Rotary Position Embedding

   Rotary position embeddings apply position-dependent rotations to projected queries and keys. The construction divides an even-dimensional vector into coordinate pairs and rotates each pair using its positional angle. Comparing a rotated query at position m with a rotated key at position n introduces relative displacement into their dot product: the positional contribution depends on m−n. Position therefore enters query-key compatibility through transformations of the vectors, rather than only through an added token-state vector.

5. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

   Attention lets context influence token representations, while the demonstrated decoder attention matrix prevents influence from future positions.

6. Training an LLM from Scratch, Locally

   The workshop forward pass combines token and positional embeddings, processes transformer blocks and normalization, then produces LM-head logits and a training loss.

7. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

   BERT uses a stack of bidirectional Transformer encoder layers: each token representation can incorporate supplied context on both sides. Its motivation includes token-level tasks where right-hand context matters. Predicting selected corrupted tokens supplies a training objective compatible with this visibility; unrestricted access to an unchanged prediction target would permit leakage. Different output heads use the resulting states differently: classification uses the final state of the added CLS token, while extractive question answering predicts answer boundaries within the supplied passage. Bidirectional representation building therefore need not produce an autoregressive continuation.

8. Optimizing LLMs for Speed and Memory

   Model parameters are numerical weight matrices and vectors loaded from a checkpoint; text inputs are represented separately as sequences of vectors. In ordinary inference, request inputs pass through those weights without a training update. Applied to RAG, instructions, conversation history, the question and retrieved text belong to request context when included in the input. Changing that text changes the computation without rewriting the checkpoint. Request-specific cached attention keys and values are intermediate computation state, not newly learned model parameters.

9. PyTorch Embedding

   An embedding is an indexed table with learnable shape vocabulary-size by embedding-width. Integer inputs select rows; an input of shape n produces output of shape n by d. A vector here is a list of d numerical coordinates, and stacking token vectors forms a matrix. Repeated indices select the same row before subsequent positional and contextual processing. The integer serves as an address, rather than a feature magnitude.

10. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    The demonstrated GPT-2 implementation obtains a token embedding by selecting the corresponding row of the learned model_wte matrix.

11. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    Embeddings are learned with the model's other parameters through prediction training rather than manually assigned semantic coordinates.

12. Linear — PyTorch documentation

    A linear layer transforms the final feature dimension while preserving preceding dimensions. Its learned weight and optional bias are separate from the supplied input. Writing the documented operation as Y=XW+b, an n-by-d input and d-by-r matrix produce an n-by-r output. Each output coordinate is the sum of corresponding input and weight products, plus bias. That sum is a dot product. The same transformation applies independently to every sequence row; this projection alone does not exchange information between positions.

13. Sequence to Sequence Learning with Neural Networks

    Ilya Sutskever, Oriol Vinyals and Quoc Le at Google addressed mappings between variable-length sequences using two recurrent networks. Recurrence carries a numerical state from one position to the next. Their encoder reads the source into a fixed-dimensional state; their decoder generates a target conditioned on that state. Their five-model ensemble exceeded the reported non-neural translation baseline on WMT 2014 English-to-French. Reversing source order improved their results, and the paper reports successful handling of long sentences. Fixed-summary models therefore should not be portrayed as uniformly failing before attention arrived.

14. Neural Machine Translation by Jointly Learning to Align and Translate

    Bahdanau, Cho and Bengio addressed the difficulty of compressing an entire source sentence into one fixed-length vector. Their recurrent encoder instead produces a sequence of contextual source representations. For each translated word, a learned alignment network compares the preceding decoder state with source representations, normalizes the scores, and forms a weighted source summary. Alignment here means a learned correspondence between target and source positions, rather than a manually supplied word mapping. Attention therefore preceded the Transformer and initially complemented recurrent computation.

15. Attention Is All You Need

    The Transformer replaced recurrent sequence traversal with attention to enable parallel computation across supplied positions and shorten paths between distant positions. Section 4 contrasts constant-length paths through unrestricted self-attention with paths growing with sequence length in recurrent networks; restricting attention to neighborhoods lengthens those paths. Section 3.2.1 also explains score scaling: assuming independent query and key coordinates with zero mean and unit variance, their dot product has variance d_k. Dividing by sqrt(d_k) counteracts this growth, which otherwise can push softmax into regions with very small gradients.

16. Attention Is All You Need

    Ashish Vaswani and colleagues' 2017 Transformer demonstrated that an encoder-decoder built without recurrent or convolutional sequence processing could achieve strong translation results. Its reported English-to-German scores exceeded the listed recurrent and convolutional comparators, including ensembles. The paper also evaluated English constituency parsing, extending the demonstration beyond translation.

17. Improving Language Understanding by Generative Pre-Training

    Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever at OpenAI investigated whether one generatively pretrained Transformer could transfer to varied language-understanding tasks with minimal architectural changes. Their system improved the reported best results on nine of twelve studied tasks. Removing pretraining hurt every evaluated task in their ablation. This establishes an early use of the causal Transformer as a reusable representation model, beyond producing continuations.

18. Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

    The paper constructs a self-attention block from multi-head attention, residual additions, row-wise feed-forward transformations and LayerNorm, without positional encoding or dropout. It establishes permutation equivariance: rearranging input elements rearranges their output representations correspondingly. This differs from permutation invariance, where rearrangement leaves the output unchanged. Its later pooling operation supplies invariance for set-level outputs.

19. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    GPT-2 adds learned position vectors to token embeddings; the supplied position table also bounds the positions supported by this implementation.

20. Attention Pooling by Similarity — Dive into Deep Learning

    Attention compares a query with keys to obtain similarity scores. Normalized scores weight the associated values; summing those weighted values gives the query-dependent output. Keys determine addressing weights, while values supply the combined content. Equation 11.2.2 explicitly separates query, keys, and values. This explains why retaining both keys and values supports subsequent attention computations.

21. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    Learning extends beyond MLP weights and biases to embeddings, attention parameters, and normalization parameters.

22. Attention Is All You Need

    Attention transforms vectors into queries Q, keys K, and values V through learned projections. Each query–key dot product measures compatibility. Dividing by √dk, where dk is key dimension, controls score scale. Row-wise softmax exponentiates and normalizes scores into nonnegative weights summing to one. The output softmax(QKᵀ/√dk)V is a weighted mixture of value vectors, one per query. In self-attention, all three projections originate from the same representation. In cross-attention, queries originate from the representation being updated, while keys and values originate from a separate conditioning representation. Their sequence lengths can differ. Multiple heads learn separate projections, then concatenate and project their outputs.

23. Effective Approaches to Attention-based Neural Machine Translation

    Minh-Thang Luong, Hieu Pham and Christopher Manning at Stanford compared global and local attention in September 2015. Global attention reads all encoder states for each target prediction; local attention forms a weighted summary within a selected source window, addressing the expense of inspecting every source position. They examined dot-product and learned bilinear compatibility functions alongside other scoring choices. They also fed the preceding attention-informed state into subsequent decoding. Their English–German experiments showed improvements from attention and further gains from this input feeding. Attention thus developed as a family of addressing and information-flow choices within recurrent models before the Transformer.

24. PyTorch scaled_dot_product_attention

    Ignoring batch and head axes, queries have shape L by d_k, keys S by d_k, and values S by d_v. Scores and weights have shape L by S; the output has shape L by d_v. Softmax runs across source positions. For square causal attention, the permitted entries include the diagonal and lower triangle; excluded scores receive negative infinity before softmax. Constructed teaching example: one query with scaled scores [0,0] and values [2,0] and [0,4] produces weights [0.5,0.5] and output [1,2]. Excluding the second source instead yields [1,0] and [2,0].

25. Low Level Technicals of LLMs

    A causal decoder must predict from preceding tokens without access to future target information.

26. Decoding Mistral AI's Large Language Models

    Instruction tuning uses prompt-response pairs and trains next-token prediction on the response while masking the prompt.

27. Low Level Technicals of LLMs

    Causal masked attention lets a shifted training sequence represent prefix-conditioned predictions while preventing access to future tokens.

28. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    T5 represents relative position with learned scalar biases added to attention logits before attention weights are computed. Bias selection depends on the offset between query and key positions. Its implementation groups offsets into buckets, shares position parameters across layers, and uses different parameters for different heads. The reported configuration uses 32 buckets, with increasingly broad ranges and a shared bucket for sufficiently large offsets. This supplies a concrete contrast with adding absolute position vectors or rotating queries and keys.

29. RoFormer: Enhanced Transformer with Rotary Position Embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen and Yunfeng Liu at Zhuiyi Technology introduced RoFormer in April 2021. The initial paper supplied preliminary Chinese experiments rather than the English evaluations present in later revisions. Its legal-case matching task asked which of two case descriptions better matched a third. RoFormer performed comparably to WoBERT at a 512-token cutoff and improved when its cutoff increased to 1,024 tokens.

30. Low Level Technicals of LLMs

    RoPE is presented as encoding token position through sine/cosine-weighted rotations, with the Roformer paper named as its reference.

31. Low Level Technicals of LLMs

    Insufficient position precision can collapse distinct RoPE positions into the same value.

32. Are Sixteen Heads Really Better than One?

    The authors ablate attention heads in a translation Transformer and a fine-tuned BERT model. Many individual heads can be removed with little performance change, and some layers retain comparable performance with a selected single head. Other components, particularly some encoder-decoder attention layers, depend substantially on multiple heads. This demonstrates that the presence of multiple heads does not establish that each contributes a unique, indispensable function.

33. Deep Residual Learning for Image Recognition

    A residual connection adds a learned transformation to its input: y=x+F(x). The shortcut supplies an identity path, while the transformed branch learns a correction relative to that input. Elementwise addition requires matching dimensions; the paper introduces projected shortcuts when dimensions differ. Its motivation is that learning a residual correction can make optimization easier than learning the complete mapping, supported by experiments on deep image-recognition networks.

34. On Layer Normalization in the Transformer Architecture

    A pre-normalization block can be written u=x+Attention(LayerNorm(x)), followed by y=u+FFN(LayerNorm(u)). Normalization precedes each transformed branch while the residual path carries the unnormalized accumulated state. The paper also applies final normalization before prediction. Post-normalization instead normalizes after each sublayer's residual addition. LayerNorm calculates mean and standard deviation over coordinates of an individual representation vector and applies learned scale and bias. These blocks preserve the outer sequence-by-model-width shape.

35. nanoGPT model.py

    nanoGPT's forward method adds learned token and position embeddings, executes blocks sequentially, and applies final LayerNorm. With targets, its vocabulary head scores every position and computes cross-entropy; without targets, it projects only the final position. Its separate generation method repeatedly calls forward, takes final-position logits, applies vocabulary softmax, samples an ID and appends it to the prefix. No optimizer update occurs in this loop. Mechanism inference: known input positions are processed together inside each layer, whereas subsequent generated inputs depend on earlier selections. The model's width-d to width-4d to width-d MLP also makes its per-position work explicit.

36. Root Mean Square Layer Normalization

    RMSNorm rescales a vector using its root mean square: the square root of the average squared coordinate. It then applies learned coordinatewise gains. Unlike LayerNorm, it does not subtract the coordinate mean. The paper separates rescaling from recentering and proposes RMSNorm as a simpler normalization operation that retains rescaling invariance.

37. LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron and colleagues at Meta AI documented a concrete later combination of Transformer components in February 2023. LLaMA normalizes sublayer inputs using RMSNorm and uses rotary position embeddings at each layer instead of absolute positional embeddings. The paper explicitly credits Su and colleagues for RoPE. These choices provide a documented downstream use of rotary positions and show that normalization placement and positional handling can change while the Transformer block structure remains recognizable.

38. Training an LLM from Scratch, Locally

    A residual update adds attention's output to the existing activation instead of replacing that activation.

39. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    GPT-2 Small repeats the same block operations twelve times, but each block has its own parameters.

40. Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code

    Residual connections let information persist around attention and perceptron transformations, supporting a communication-bus view of cooperating layers.

41. Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code

    Logit lens applies the language head between layers to reveal how vocabulary rankings evolve before the final prediction.

42. Improving Language Understanding by Generative Pre-Training

    The original GPT uses a decoder-only Transformer: token and position embeddings pass through repeated masked self-attention and position-wise feed-forward blocks, followed by a vocabulary projection and softmax. The mask restricts a position to earlier context instead of attending to future tokens. Unlike the original encoder-decoder translation Transformer, this language model does not require a separate source encoder and encoder-decoder attention path. Training optimizes prediction of successive tokens; inference applies the learned parameters to the supplied sequence.

43. Attention Is All You Need

    The original decoder combines masked target self-attention, encoder–decoder attention and a position-wise feed-forward sublayer. Section 3.2.3 specifies decoder-side queries and encoder-output keys and values for encoder–decoder attention, allowing each target position to attend across the source sequence.

44. Attention Is All You Need — Figure 1 decoder architecture

    Figure 1 shows each original decoder layer proceeding upward through masked multi-head self-attention, encoder–decoder multi-head attention, and a feed-forward sublayer, in that order. An Add & Norm block follows each sublayer. The encoder stack feeds the middle attention sublayer; the decoder state reaches it after masked self-attention and its Add & Norm block.

45. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google tested BERT's bidirectional design against a left-context-only language model using matched pretraining data, fine-tuning procedures and hyperparameters. Without next-sentence prediction in either variant, the bidirectional masked-language model performed better on all five reported development tasks, especially passage-answer extraction and paraphrase classification. Adding a bidirectional recurrent layer during fine-tuning improved answer extraction but did not recover the bidirectionally pretrained model's result.

46. The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

    Supporting BERT, Qwen, and ModernBERT requires accounting for architecture-specific forward-pass behavior rather than assuming identical attention and position handling.

47. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    The demonstrated language head scores vocabulary tokens using the token embedding matrix, then chooses the largest score for deterministic comparison with another GPT-2 implementation.

48. Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code

    The language head applies LayerNorm and an unembedding matrix to produce logits, then the spreadsheet selects the highest-ranked token for its temperature-zero output.

49. Text generation — Hugging Face Transformers

    Autoregressive generation repeatedly predicts a next token using the prompt and tokens already generated. In notation, P(y|x)=product_t P(y_t|x,y_<t). Decoding chooses each token, for example by taking the highest-probability token or sampling, then extends the sequence. Generation stops on a configured stopping condition such as an end token or output-length limit. Transformers exposes max_new_tokens to bound newly generated tokens; decoder-only generation returns prompt tokens plus the continuation.

50. Transformers T5 documentation: Training

    Teacher forcing supplies the recorded target sequence as decoder context, shifted so that each position predicts the following target. The documented T5 procedure prepends its start token to decoder inputs and uses target tokens followed by EOS as labels. This establishes the distinction between conditioning training predictions on recorded predecessors and feeding generated choices back during generation. Padding tokens are artificial entries used when batching sequences of different lengths.

51. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    Autoregressive generation repeatedly appends a predicted token to the input and predicts again.

52. Transformers GenerationConfig: stopping and output budgets

    Generation can end on an end-of-sequence token, a configured stop string, a length limit, or other stopping criteria. max_new_tokens limits generated tokens independently of prompt length; max_length includes the prompt. These are upper bounds, not promises to produce that many tokens. max_time is not a hard deadline: generation finishes its current pass after the allotted time. For ordinary single-beam generation, do_sample selects sampling versus greedy decoding. An application should distinguish normal completion from length truncation, cancellation and failure rather than interpreting every returned prefix as a completed answer.

53. Transformers: why causal KV caching works

    Under causal attention, appending future tokens does not change previously computed token representations. Each layer can therefore retain past keys and values. At decoding step t, it computes the new token's query, key, and value, appends k_t and v_t to that layer's cache, and attends with q_t over the retained and current keys and values. This avoids recomputing old projections and old positions' attention outputs. It does not eliminate attention to the retained history: ordinary full-context decoding still performs work proportional to the cached sequence length per new token.

54. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

    Dense attention conceptually compares N queries with N keys, producing N-by-N scores before combining values. The paper gives an exact attention algorithm with O(N²d) arithmetic but O(N) additional memory beyond inputs and outputs at fixed head width. Consequently, doubling sequence length approximately quadruples pairwise attention arithmetic without requiring every implementation to store the complete attention matrix. Its algorithm preserves dense attention rather than removing connections.

55. Longformer: The Long-Document Transformer

    Longformer sparsifies attention by selecting which position pairs can interact. Sliding-window attention connects each token to nearby positions; stacking layers expands the receptive field, meaning the input region that can influence a later representation. Dilated windows skip positions to extend reach. Selected global positions attend across the sequence and are accessible from every position. For question answering, the paper assigns global attention to question tokens; classification uses a global CLS token. With fixed window width and a fixed number of global positions, these patterns require interactions growing linearly with sequence length.

56. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    The talk distinguishes quadratic attention computation from linearly growing sequence-dependent memory; solving one does not eliminate the other.

57. Transformers: cache allocation versus retained history

    KV caches retain attention keys and values beyond model weights. DynamicCache grows as tokens arrive; StaticCache preallocates a maximum capacity, masking unused positions and potentially wasting memory and attention work on short sequences. Sliding-window or chunked layers stop growing at their retained-history limits even when the configured sequence capacity is larger. Offloading transfers most layer caches through CPU memory; quantized caches reduce precision but can increase latency for short contexts. Operational inference: allocated slots are a storage decision, not evidence that the model supports that many sequence positions. For ordinary full-history static caching, reserve sufficient slots for prompt plus intended output while independently enforcing the model's supported context limit. Windowed cache capacity must instead follow the architecture's retention rules.

58. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

    ALiBi adds a distance-proportional penalty to query-key scores before softmax, using a fixed slope for each head rather than learned position embeddings. This favors nearer positions without forbidding distant ones. Controlled WikiText-103 experiments trained models on 512- or 1,024-token sequences and evaluated longer inputs. Positional methods differed substantially in how their next-token predictive fit changed beyond training length; ALiBi maintained stronger extrapolation in the reported comparisons. A positional formula being computable at unseen indices is therefore insufficient evidence of reliable length generalization.

59. Extending Context Window of Large Language Models via Position Interpolation

    Position Interpolation addresses extending an already pretrained RoPE model. It rescales position indices into the original positional range before applying rotary encoding, then fine-tunes the model to the changed representation. The reported LLaMA experiments extend a 2,048-token window to as much as 32,768 tokens and evaluate language modeling, passkey retrieval and summarization. This is adaptation through changed positional handling and further training, distinct from simply evaluating an unchanged checkpoint beyond its training length.

60. Lost in the Middle: How Language Models Use Long Contexts

    The paper tests multi-document question answering and synthetic key-value retrieval while controlling context length and where relevant information appears. Reordering the answer-containing document changes its position without changing the desired answer. Several evaluated models performed better with evidence near the beginning or end than in the middle; results varied by model and task. This separates fitting tokens inside a supported context window from reliably retrieving and using their information. It motivates controlled position and distractor tests rather than assuming nominal length measures effective use.

61. RULER: What's the Real Context Size of Your Long-Context Language Models?

    RULER varies sequence length and task complexity using retrieval, variable-chain tracing, aggregation and question answering with distractors. A needle-in-a-haystack task retrieves a specified item from surrounding irrelevant material; RULER extends this beyond single-item retrieval. In its evaluated models, strong simple-retrieval performance often coexists with deterioration on more demanding tasks as inputs lengthen. Accepting a sequence and successfully using its information are therefore distinct claims.

62. Attention is Not Only a Weight: Analyzing Transformers with Vector Norms

    An attention head's output can be expressed as a sum of attention weights times value vectors transformed by the output projection. A large weight can therefore multiply a very small vector and make only a small additive contribution. In the paper's BERT analysis, special tokens and punctuation often received large attention weights while their transformed-vector contributions were comparatively small. Examining the vectors changed the interpretation suggested by the attention heatmap alone.

63. Attention is not Explanation

    Sarthak Jain and Byron Wallace's June 2019 paper tested whether attention weights identify inputs responsible for predictions. Across classification, question answering and inference tasks, weights in their recurrent models correlated weakly or inconsistently with gradient-based importance and changes caused by removing tokens. They also constructed substantially different attention distributions that preserved nearly identical predictions while other model parameters remained fixed. Their findings challenge reading a heatmap directly as an explanation of a prediction.

64. Attention is not not Explanation

    Wiegreffe and Pinter test attention interpretation using uniform-attention baselines, variation across random initializations, transferred attention distributions and adversarially trained alternatives. Learned attention offered little benefit on some classification tasks, while attention distributions transferred from trained recurrent models improved a separate token-level classifier over uniform weighting on the tested datasets. Their findings support testing whether an attention interpretation is useful in a specific setting instead of treating either a heatmap or its manipulability as a universal verdict.

65. Low Level Technicals of LLMs

    Compare intermediate layer outputs against a reference implementation using a log L2 error plot, but establish the reference before automating variant searches.

66. Training an LLM from Scratch, Locally

    Each transformer block has its own weights, attention, normalization, and MLP, also called an FFN.

67. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    Random sampling is an added selection step; the speaker contrasts it with greedy selection, also described as temperature zero.

68. $1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpentero

    Unpadding, sequence packing, and FlashAttention address different sources of overhead: meaningless padding work and expensive attention-memory traffic.

69. $1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpentero

    ModernBERT's alternating attention is presented as a way to capture localized attack signals while retaining broader context without using global attention in every layer.

70. Why More Context Makes Your Agent Dumber and What to Do About It

    The speaker reports a U-curve pattern in code-review experiments: initial goals and final inputs receive attention while intervening context can be neglected.

71. Training an LLM from Scratch, Locally

    For the creative-text exercise, sample from token probabilities rather than always choosing the highest-scoring token, and use top-k to restrict the candidate set.

72. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    DeepSpeed Ulysses context parallelism distributes attention heads across GPUs while retaining full-sequence attention for each assigned head.