Transformers & attention

What a transformer computes

Interpreting a sequence requires more than identifying its individual elements. In a sentence, nearby descriptions can change what a verb implies. A model therefore needs a contextual representation: a numerical state that depends on an element and the information available at its position. Attention builds such states by combining information across positions with weights that depend on the input. [5]

For text, the starting elements are tokens, sequence units identified by integers. A token need not be a whole word; Tokenization explains how text becomes these IDs. A transformer converts the IDs into vectors, repeatedly transforms those vectors, and passes the resulting states to an output component. A language model produces scores for possible next tokens. Other models use contextual states for classification or locating an answer inside a passage. [6][7]

A forward pass applies the model's operations to supplied inputs. During ordinary inference, its fitted parameters stay fixed while the intermediate states change with the input. Changing a passage can therefore change a prediction without retraining. This distinction will recur throughout the chapter: parameters specify learned transformations; the current sequence supplies the information those transformations process. [8]

The arrays and their parameters

A vector is an ordered array of numerical coordinates. An embedding table stores one learned vector per vocabulary entry. If the vocabulary contains $v$ entries and each vector has $d$ coordinates, the table $E$ has shape $v\times d$. A token ID selects a row; the magnitude of the ID has no feature meaning. Stacking the selected vectors for $n$ sequence positions produces a matrix $X$ of shape $n\times d$. [9]

Repeated IDs initially select identical rows. Their later states can differ once position and context enter. The table contains parameters, fitted numbers shared across inputs; the selected and transformed arrays are activations, intermediate values computed for this input. Their learned coordinates are not manually assigned labels such as tense or importance. Parameters and modeling assumptions explains the shared-parameter distinction; Vectors and coordinate spaces develops representation geometry. [11][8]

A linear projection forms new coordinates from weighted sums of existing ones. A dot product multiplies corresponding coordinates and sums them: [1, −1] · [2, 3] = 2 − 3 = −1. In $Y=XW+b$, an $n\times d$ input and $d\times r$ weight matrix yield an $n\times r$ output; $b$ adds a learned offset. The same transformation applies separately to every row. It changes features without exchanging information between positions. Here weights use the mathematical $XW$ convention; libraries may store them transposed. [12]

Why attention emerged

Early neural translation systems made the information-access problem concrete. An encoder processes a source sequence; a decoder produces a target sequence. In a recurrent network, a numerical state passes from one position to the next. A decoder could receive a fixed-dimensional summary of the source, but that made the summary responsible for preserving everything subsequent predictions might need. [13][14]

The useful thread is adaptive information access. Attention first complemented recurrence, then became the central interaction mechanism of a different architecture. Its matching rule, permitted sources and output use remain separate design choices.

Supplying sequence order

Array positions are visible to your program, but they do not automatically become features in its numerical transformations. With unrestricted attention and no positional signals, rearranging the input rows rearranges their output representations correspondingly. This is permutation equivariance. It does not mean the output array remains unchanged. The statement also does not cover arbitrarily rearranging tokens while holding a causal visibility pattern fixed. [18]

One solution adds a position vector to each token vector. An absolute position is an index within the sequence. GPT-2 uses a learned position table distinct from its token table: position selects a row from one, token identity selects a row from the other, and their coordinates are added. [19]

Position vectors can also be fixed: the original Transformer added sine/cosine patterns at different frequencies. GPT-2 instead learned them. Both supply position; neither addition determines permitted attention connections. [19]

Matching and transmitted content

Attention separates deciding where to read from deciding what to transmit. A query scores candidate information. A key is the representation compared with that query. A value supplies content to the resulting mixture. Keys and values stay paired by source position, but serve different roles. [20]

Self-attention projects the same sequence states separately into Q, K and V. Projection parameters are shared across inputs; these activations depend on the input. Omitting optional biases: [21][22]

Different projections make permitted query–key comparisons directed: q_i · k_j need not equal q_j · k_i. The names denote numerical roles, not questions or database identifiers. [22]

Matching functions were already a design choice in recurrent attention. Effective Approaches to Attention-based Neural Machine Translation, by Minh-Thang Luong, Hieu Pham and Christopher Manning at Stanford in September 2015, compared dot-product and learned compatibility functions, alongside global and local source access. Its improvements also involved feeding attention-informed states into subsequent decoding. Transformer attention adopts a particular matching-and-aggregation construction from this broader design space. [23]

Computing the attention update

Each query–key dot product produces a compatibility score. Softmax exponentiates a row of scores and divides each result by their sum, yielding positive weights totaling one. Those weights multiply the corresponding value vectors, which are summed into one output vector per query. Attention therefore produces a mixture, rather than selecting only the highest-scoring source. [20][5]

Dividing scores by $\sqrt{d_k}$, the square root of query/key width, counters variance growth under independent, zero-mean, unit-variance coordinates, avoiding sharply concentrated softmax weights with little sensitivity during learning. [15]

For a numerical example, take scaled scores [0, 0] and values [2, 0] and [0, 4]. Exponentiation gives [1, 1]; normalization gives [0.5, 0.5]. The output is 0.5[2, 0] + 0.5[0, 4] = [1, 2]. Increasing one score redistributes weight across the row. The weights allocate content; they do not express confidence that the content is true. [20]

Restricting information access

An attention mask specifies permitted query–source pairs. In causal self-attention, a position may read itself and earlier positions, but cannot read later ones. Its state can then predict the next token without seeing that target. A small learned attention weight is not a prohibition: the architecture must exclude the forbidden connection. [1][25]

Padding adds artificial fillers when batching unequal sequence lengths. A padding mask excludes those source positions; a causal mask excludes future connections. Combine the exclusions when both apply. Source masking does not itself discard padded query outputs or their losses. Boolean conventions also differ: True means allowed in PyTorch's scaled-dot-product attention interface, but excluded in MultiheadAttention's Boolean masks. [2][24]

Keep three decisions separate: position describes placement, attention masking controls access, and loss selection determines which predictions receive a training penalty. Removing a prediction from the loss does not stop another position from reading its input. Separate visibility from scoring develops this distinction. [26][27]

Relative position in matching

Position can affect compatibility directly. Relative position describes the displacement between query and source indices. In T5, a learned scalar bias selected by that displacement is added to the attention score before softmax. Offsets are grouped into buckets, so multiple distances can share a bias. This differs from adding a vector to the initial token state: the positional term modifies a relationship between two positions. [28]

Rotary Position Embedding, or RoPE, rotates pairs of projected query/key coordinates. Jianlin Su and colleagues introduced it in RoFormer in April 2021. For column vectors q and k, let R_m rotate a coordinate pair according to position m: [4][29]

These mechanisms enter at different locations: absolute vectors alter initial states, scalar biases alter scores, and RoPE alters the vectors being compared. They still feed a weighted-value computation. Numerical precision matters in that path: Daniel Han describes a reduced-precision implementation in which distinct large position indices collapsed to the same value, losing positional distinctions. A mathematically suitable representation still needs faithful numerical execution. [30][31][28]

Several mixtures in one layer

Each attention head separately projects Q/K/V and computes weights. Concatenating h mixtures of shape $n\times d_v$ gives $n\times(hd_v)$; learned projection restores width d. Conventionally, $d_v=d/h$. [22]

The same sequence positions participate in every head. What varies is their projected representation and resulting allocation of attention. Heads therefore offer multiple learned interaction patterns within one layer; they are not separate complete language models. In Ishan Anand's spreadsheet demonstration, one head shows an interpretable pronoun–antecedent pattern, while others lack such an obvious reading. A head receives no guaranteed linguistic job title. [1]

The study Are Sixteen Heads Really Better than One? removed heads from trained translation and BERT models. Many removals had little effect, while some encoder–decoder attention layers relied substantially on multiple heads. Head count alone therefore does not establish how many indispensable functions a model learned. Pruning a trained model also differs from training a smaller model from the outset. [32]

The complete block

Attention supplies an update, but a block also needs to preserve and transform the state receiving it. A residual connection adds a branch's result to its input: $y=x+F(x)$. Matching dimensions make this elementwise addition possible. The identity path lets the branch learn a correction to an existing representation. It neither guarantees a small correction nor skips execution of the branch. Deep Residual Learning established this construction in deep image networks. [33]

Layer normalization, or LayerNorm, controls feature scale separately at each position: it subtracts the vector's coordinate mean, divides by its standard deviation with a numerical safeguard, then applies learned scale and bias. Pre-normalization places this operation before each transformed branch; post-normalization places it after residual addition. One pre-normalization block is: [34]

A feed-forward network, or FFN, transforms each position independently using shared parameters. A common version, also called a multilayer perceptron or MLP, applies two affine transformations separated by a nonlinear activation. The intervening nonlinearity prevents the pair from collapsing into one affine transformation. In nanoGPT, feature width expands from d to 4d and returns to d. Attention mixes positions; this network processes features at each position. [35]

Normalization variants change the branch computation. Root mean square normalization, or RMSNorm, divides by a scale derived from the average squared coordinate and applies learned gains; unlike LayerNorm, it does not subtract the mean. LLaMA, documented by Hugo Touvron and colleagues at Meta AI in February 2023, combines input normalization using RMSNorm with RoPE. This is a concrete block variant, not evidence that either choice alone explains its performance. [36][37]

When reading code, locate both addition points and inspect what each branch receives. Training an LLM from Scratch, Locally makes the central change explicit: replacing X with an attention result differs from adding that result to X. [38]

Context across layers

Stacking blocks composes learned transformations. Later queries, keys and values come from states that earlier blocks have already contextualized. Blocks repeat an operation pattern, but normally have distinct parameters. GPT-2 Small, for example, uses twelve separately parameterized blocks; depth does not mean repeatedly applying one shared set of weights. [39][21]

Consider a permitted dependency: position 0 contributes to position 1 in one layer, then position 1 contributes to position 2 in the next. The second interaction can transmit information originating at position 0. Under a causal mask, each edge travels from an equal or earlier source position, so composing edges cannot create access to a future token. Feature transformations and residual additions process what those interactions deliver. [40][25]

The accumulated per-position state is often called the residual stream. For the same token at the same position in two different prefixes, its initial lookup vector can remain identical while subsequent states differ. This input sensitivity under fixed parameters enables prompting and in-context learning. A token state is also distinct from a deliberately constructed whole-item embedding. [40][5][8]

Depth and generation count are separate dimensions. Passing through another block refines representations within one forward pass; it does not append another token. An inspection technique called a logit lens applies the model's output mapping to intermediate states, turning them into scores for candidate vocabulary tokens. This reveals how candidate rankings change across layers, but does not establish a universal sequence of linguistic stages or a complete causal explanation. [41][39]

Three architecture families

Cross-attention matches target queries to source keys/values: target-by-source weights. The original decoder orders masked self-attention → cross-attention → FFN, each followed by residual addition and LayerNorm. [43][44]

A decoder-only model such as early GPT removes the need for a separate source encoder and cross-attention path. Its causal blocks build representations for predicting successive tokens. The original GPT work also used these representations for supervised downstream tasks, showing that a causal architecture was useful beyond producing free-running continuations. [42][17]

BERT builds states from both left and right context. Its added classification token, CLS, supplies a final state for classification; another output head predicts answer boundaries in a passage. Pretraining predicts selected corrupted tokens, so this visibility need not expose unchanged answers. Compared with a left-context-only model, BERT improved the reported development-task results, but the experiment changed both visibility and objective. Changing a trained model's mask alone would not reproduce it. [7][45]

Architecture identifies the information path, not every interface convention. Models sharing a family can require different chat templates, normalization operations and positional handling. A reusable runtime must preserve the computation expected by each checkpoint. [46]

Reading out vocabulary scores

An output head maps contextual states to task predictions. For next-token generation, the language-model head reads the state at the last input position, after the final block and normalization, and projects it to one score per vocabulary entry. These unnormalized scores are logits. The state supplies features for this scoring operation; choosing a token from the scores is a separate step. [47][48]

Some models tie the output weights to the input embedding table. In GPT-2's JavaScript walkthrough, the final state is dotted with each embedding row. This is often called unembedding, but it does not invert the preceding contextual transformations. The demonstration selects the largest logit; softmax would preserve that ranking. [47]

These probabilities describe possible next tokens under the model. They do not certify that a statement assembled from those tokens is factually supported. [49]

Predicting recorded successors

During training, a recorded sequence supplies reference predecessors and correct successors. Teacher forcing means conditioning a prediction on those recorded predecessors rather than on the model's earlier sampled choices. A loss penalizes predictions against their targets. The model can calculate many such predictions together because their reference inputs are already known. [50][27]

B is both position 0's target and position 1's input. That is safe because position 0 cannot read position 1, directly or through later layers. Target shifting establishes which answer belongs to each prediction; causal masking separately enforces the information boundary. Omitting either can produce the wrong training problem. [3][27]

Positions can be processed together within a layer, while the next layer still depends on the preceding layer's results. Loss computation and parameter updates are subsequent operations. A token omitted from loss can still supply context—for example, instruction tuning can score response tokens while retaining the prompt as input. Build targets from text explains objective construction beyond this alignment. [27][21][26]

Producing an unknown continuation

Autoregressive generation repeatedly predicts from an existing prefix, selects a token, and appends it. Greedy selection chooses a maximum-probability token; sampling draws according to a selection distribution. The chosen token becomes part of the next input, so later predictions depend on earlier selections. [49]

For a symbolic example, a model receives [A, B] and a selection yields C. The next prediction receives [A, B, C]; suppose its selection yields D. The second input could not be assembled before C was chosen. Training instead already has the reference C. Both processes use the same kind of forward computation, but only training performs a parameter update. [51][50][35]

Generation ends when a stopping condition is met, such as an end-of-sequence token, a configured stop string or an output-length limit. A length cutoff can interrupt an unfinished answer; stopping does not certify completeness or correctness. Selection and stopping covers decoding controls, while Reconstructing generated text explains converting IDs back into text. [52]

A key–value cache retains per-layer K and V activations. Under causal attention, appending tokens cannot change earlier states, so their projections can be reused when the prefix, parameters and positional treatment remain compatible. Processing the newly appended token computes its Q/K/V, adds its K/V to the cache, and uses its query to read retained history. Caching avoids recomputing earlier states; ordinary full-history attention still does work over the retained sources. See What the KV cache retains. [53]

Expand ⤢

Reach, work and storage

Dense attention compares every query with every permitted source. For N positions, unrestricted attention has N² pairs; diagonal-inclusive causal attention has N(N+1)/2. Both grow quadratically. At fixed feature widths, projections and per-position feed-forward work instead grow linearly with position count. These are operation counts, not predictions of complete-model latency. [54][35][27]

Longformer combines local windows, restricting reads to neighbors, with global positions that read and are readable everywhere. Layers expand the receptive field—inputs influencing a state. Fixed window width and global-position count give linear interaction growth, without reproducing dense attention. This bidirectional pattern needs modification for causal decoding. [55]

FlashAttention preserves dense attention while avoiding storage of the complete score matrix. Its exact algorithm retains quadratic arithmetic, with linear additional memory at fixed head width; floating-point results can differ. Restricting connections and changing storage are therefore different interventions. [54]

If a fixed checkpoint must preserve dense outputs and arithmetic is affordable, removing connections changes the computation you intended to preserve. An implementation that avoids large intermediates addresses the storage problem directly. Online state and tiled attention explains that mechanism. Execution limits and Road to 5 Million Tokens connect sequence growth to practical bottlenecks. [56]

Expand ⤢

When longer context helps

A context window describes a supported sequence allowance. Counting complete requests explains what consumes it. Allocating additional cache slots does not extend a model's positional support. Conversely, a windowed layer can retain only recent states while processing a longer overall sequence. Storage capacity and sequence support answer different questions. [57]

Length extrapolation evaluates beyond training lengths without further adaptation. Attention with Linear Biases, or ALiBi, adds a distance penalty with a fixed slope per head before softmax. Its controlled WikiText-103 experiments showed stronger extrapolation than the compared positional methods. The measure was perplexity—next-token predictive fit—not arbitrary answer accuracy. A position formula being computable at an unseen index does not establish reliable use there. [58]

Position Interpolation instead rescales positions into a pretrained RoPE model's original range and then fine-tunes it. Its historical LLaMA experiments extended a 2,048-token window to as much as 32,768, evaluating language modeling, retrieval and summarization. This was adaptation, not unchanged-checkpoint extrapolation; some original-window benchmarks degraded. Extending the accepted range and retaining earlier capabilities both need assessment. [59]

Even within an accepted window, available evidence can be used unevenly. Lost in the Middle moved relevant information within multi-document question-answering and key-value retrieval inputs. Several tested models performed better with evidence near the beginning or end than in the middle, with results depending on model and task. The controlled variable was evidence placement; this was not proof of a universal position curve or of literal deletion of middle tokens. [60]

RULER extends single-item retrieval tests to variable-chain tracing, aggregation and question answering with distractors. In its evaluated models, success at simple retrieval could coexist with deterioration on harder long-context tasks. Effective context length therefore needs a task, scoring rule and performance threshold. Test the operations your application requires, with workload coverage and input arrangement treated explicitly. [61]

Interpreting attention patterns

An attention map displays mixing coefficients for one input, head and layer. Contribution also depends on the values and output projection. For a scalar example of transformed contributions, weights [0.9, 0.1] multiplying values [0.01, 10] yield contributions [0.009, 1]. The smaller weight contributes more. Attention is Not Only a Weight found analogous distinctions in BERT: some heavily attended special tokens and punctuation had comparatively small transformed-vector contributions. Neither contribution size nor coefficient size alone establishes the final prediction's cause. [62]

Attention is not Explanation, by Sarthak Jain and Byron Wallace in 2019, challenged direct interpretation of attention weights. In recurrent-model experiments, weights aligned inconsistently with gradient or token-removal measures, and substantially different attention distributions could preserve nearly identical predictions. These results concern the studied models and tests; they do not directly measure modern Transformer stacks. [63]

Attention is not not Explanation, Sarah Wiegreffe and Yuval Pinter's 2019 response, examined uniform baselines, variation across initializations and transferred attention distributions. Their recurrent-model results supported testing whether an interpretation is useful in a particular setting rather than accepting a universal verdict. The disagreement helps distinguish a visible pattern from a carefully specified explanation claim. [64]

An ablation removes or disables a component and measures the resulting behavior. Head-removal experiments test dependence under that intervention and on those inputs. They do not assign an immutable purpose to the head. [32]

For implementation diagnosis, controlled dependencies can be more decisive than a heatmap. With identical parameters, earlier tokens and positions, changing only a future token should not change an earlier causal prediction beyond numerical tolerance when randomness is disabled. An observed change justifies investigating an unintended information path. Comparing intermediate outputs layer by layer can locate where implementations first diverge; changing the sampling rule or which targets receive loss does not repair that forward-pass dependency. [25][65]

Overview

Language models process sequences of vocabulary identifiers rather than written words directly. Tokenization maps text into those units; decoding maps generated units back to text. The mapping affects what fits in a request, how languages and code are represented and how applications locate pieces of the original input. This chapter follows characters and bytes through segmentation, vocabulary lookup and chat formatting to the sequence the model actually receives.

Tokens and vocabulary IDs

A token is one entry in a tokenizer’s finite vocabulary. Encoding converts text into an ordered sequence of token IDs; decoding maps IDs back toward text. Depending on the tokenizer, an entry may represent a word, a word fragment, punctuation, whitespace, a byte sequence, or a structural marker. A token is therefore neither reliably a word nor reliably a character. The tokenizer associated with the selected model defines the mapping. [73][74]

One string, seven vocabulary entries

Repeated pieces reuse the same IDs, but the numbers themselves are names, not measurements. ID 220 is not semantically “between” IDs 17 and 489. The model uses each ID to retrieve a learned vector and then computes with those vectors; Embeddings and Representation Learning explains that numerical representation. Changing IDs or using another tokenizer changes which learned rows the model receives, so tokenizer compatibility is part of the model interface rather than an interchangeable preprocessing preference. Ishan Anand’s GPT walkthrough provides an implementation-oriented view of this transition from pieces to IDs and vectors. [76][77]

Characters, code points and bytes

Before inspecting a token boundary, name the text unit. A Unicode code point is a numbered position in the Unicode codespace. A grapheme cluster is a programmatic approximation to a user-perceived character and may contain several code points. UTF-8 then represents Unicode scalar values using one to four bytes. Some programming environments expose UTF-16 code units as yet another length and offset system. [78][79][80]

Two representations of é

A tokenizer may preserve this distinction, normalize it away, or split the byte sequences in different places. Consequently, a UI showing one apparent character does not establish a string length, byte length, token count, or safe slicing boundary. [83][79]

Preprocessing sets boundaries

Tokenizer implementations commonly separate several responsibilities. Normalization transforms text under chosen equivalence rules. Pre-tokenization establishes regions within which the segmentation model may operate. The model then selects vocabulary pieces, and post-processing may insert special tokens. These stages are configurable; they are not universal properties of tokenization. [84]

Normalization can discard distinctions an application needs, while lowercasing, accent removal, and whitespace cleanup are separate transformations. Treat every configured transformation as part of the input contract. [84]

Pre-tokenization constrains later choices. In the inspected tiktoken 0.11.0 definition, one numeric expression creates regions of one to three numeric characters; for constructed ASCII input, 1234 first becomes 123|4. Those are preliminary regions, not a claim about the final token IDs: vocabulary merges still decide the pieces inside each region. [85]

Vocabulary size, coverage and length

A whole-word vocabulary is compact for frequent known words but needs an entry for every retained form. Unseen names, misspellings, compounds, and inflections then require an unknown marker or another fallback. An unknown token preserves only the fact that some unsupported input occurred; it cannot reconstruct which input was lost. Character and byte representations avoid a huge word inventory, but usually produce longer sequences. Subword tokenization uses reusable fragments between those extremes, without promising that the fragments are linguistic morphemes. [86][87][74]

Granularity moves costs rather than removing them

Vocabulary size also affects model parameters because token IDs index an embedding table with roughly vocabulary size times vector width entries. That matters especially for tiny models: a vocabulary suitable for a large multilingual system can dominate a small educational model’s parameter budget. But a smaller inventory may increase sequence work, so neither vocabulary size nor token count alone identifies the cheapest design. Byte-based models remain a live alternative: the 2022 ByT5 study removed the learned subword vocabulary, accepted longer sequences, and reported task-dependent advantages on tested noisy and spelling-sensitive workloads. It was a separately trained architecture, not a replacement tokenizer for an existing checkpoint. [91][88]

Must-know turning points

Several lines of work contributed different parts of today’s tokenizer interface: compression, open-vocabulary segmentation, neural subword translation, and reproducible raw-text processing. [92][93][94][95]

These are connected developments, not a ladder on which each new method eliminated the previous one. Word, character, byte, BPE, WordPiece, and Unigram approaches continue to coexist because their coverage, sequence length, preprocessing, and implementation tradeoffs differ. [88][89]

Learning and applying BPE

Byte-pair encoding (BPE) separates vocabulary training from request encoding. Training repeatedly merges selected adjacent units and records their priorities. Encoding later applies those fixed priorities to new text; it does not recount the request or retrain the vocabulary. [96][97]

Counts follow the current segmentation

In the published word-bounded BPE procedure, adjacent-pair counts are weighted by word frequency. Each merge changes the current segmentation, so the next pair is counted from that updated representation. Boundary and tie policies must be specified for a reproducible vocabulary. [98]

Rank priority is not longest match

Applying ranks is not the same as greedily taking the longest vocabulary entry. A tokenizer must be identified by more than its vocabulary list. In the comparison, BPE changes merge priority; WordPiece changes the available prefix pieces; Unigram changes piece probabilities and compares their products across complete segmentations. These controls belong to different algorithms. Anand’s implementation walkthrough demonstrates the repeated-merge intuition in an accessible model reconstruction. [96][99][100][101]

WordPiece, Unigram and SentencePiece

Reusable pieces do not imply one selection algorithm. The original BERT WordPiece encoder works left to right, choosing the longest vocabulary match at each position. Continuation pieces use the ## prefix; its documented example represents unaffable as un, ##aff, ##able. If a remaining span cannot be matched, that implementation emits the unknown token for the whole word. [100]

A Unigram tokenizer instead assigns scores to vocabulary pieces and evaluates complete candidate segmentations. The best segmentation is the highest-scoring path, commonly found with the Viterbi algorithm; the same inventory can admit several alternatives. Kudo’s 2018 subword-regularization work sampled such alternatives during training to create segmentation variation. That does not imply that randomly changing segmentation at inference will improve an already-trained model. [101][102]

Different names answer different questions

Google’s 2016 GNMT system adopted speech-derived WordPiece to help with rare-word translation and copying, but the deployed system combined many changes; its overall results cannot be assigned to tokenization alone. SentencePiece later made raw-text preprocessing and reconstruction rules part of a portable tokenizer artifact, improving reproducibility without guaranteeing preservation of distinctions removed by normalization. [103][104][95]

What decoding preserves

A text round trip, decode(encode(text)), preserves the original string only when the full pipeline is reversible for that input. Normalization may replace the original representation; unsupported input may become an unknown token; decoding may omit special tokens or clean spaces. Even when text round-trips, the reverse encode(decode(ids)) need not reproduce the same IDs because several token sequences can decode to the same text. [105][106][107]

Reconstruct bytes before characters

A byte-based token boundary can fall inside a multibyte UTF-8 character. Decode the reconstructed byte stream, not each arbitrary fragment independently. An incremental decoder retains incomplete input between calls and handles any remainder when the stream ends. [80][108][106]

Do not apply that repair where it is unnecessary. Some hosted streaming APIs already deliver decoded text fragments rather than token bytes; those strings should be accumulated according to the API contract. A delivery event is not necessarily one model token. Inference Engineering covers streaming and serving behavior in depth. [109][110]

Finally, reversible tokenization does not force the model to copy text exactly. Generation selects a new sequence from model predictions. Systems that require exact identifiers, quotations, or source strings must compare the produced value with the required original or bypass generation for that field. [111][112]

Offsets need a coordinate system

An offset is incomplete without two declarations: the representation it indexes and the unit it counts. Byte offsets, Unicode code-point offsets, UTF-16 code-unit offsets, and token positions are different coordinate systems. Ranges should also state their convention; the examples here use zero-based, end-exclusive intervals. [78][82]

Name the coordinates

Normalization creates a second mapping problem: an offset into normalized text may need alignment back to the original. Hugging Face’s internal NormalizedString uses byte-level alignment data, while the Transformers fast-tokenizer interface documents returned offsets as character start/end pairs. Those contracts should not be conflated merely because both expose two integers. Applications that attach extracted values to source material need the same discipline across every transformation; Document Understanding and OCR’s alignment section develops that broader source-location problem. [113][114]

Special tokens and chat templates

A special token is a designated vocabulary entry used for structure or control. A chat template converts ordered role/content messages into the token sequence expected by a particular chat checkpoint. It can insert a beginning marker, role headers, turn endings, and—where required—a generation prompt marking the start of an assistant response. Message objects are therefore an application interface, not the representation consumed directly by the model. [115][116]

Serialization is checkpoint-specific

A pinned Mistral-7B-Instruct-v0.1 configuration formats the conversation user Hi, assistant Hello, user Bye with instruction boundaries and sequence markers. This template does not add a separate assistant prefix. Other checkpoints use different role markers and continuation rules, even when they share a base architecture. [117][115]

Three kinds of ending

Do not format twice

Apply structural formatting exactly once. If a chat template already inserts the required beginning token and a subsequent tokenization step inserts it again, the model receives a different prefix. Daniel Han’s fine-tuning bug review describes degraded inference when training and serving used different beginning-token counts. The same general rule applies to export: preserve the training serialization at inference. Padding is separate batch filler; if a training loss masks every occurrence of the padding ID, reusing an end-of-sequence ID for padding can also remove end targets from supervision. That conditional training issue belongs to Post-training and Alignment. [120][121][115][122]

Structural markers guide the model’s learned interpretation; they do not authenticate a role or authorize an action. An application must enforce those responsibilities outside the tokenizer. [116]

Language, code and formatting

A tokenizer allocates its finite vocabulary according to training data and design choices. Patterns well represented in that data may receive long reusable pieces; less represented scripts, domain terms, identifiers, and numeric forms may fragment. Consequently, comparable information can occupy different numbers of tokens. The result describes representation length under one tokenizer, not a language’s complexity or a model’s intelligence. [123][124][125]

Translated content can occupy different budgets

These corpus-specific ratios are not universal conversions. A related metric, token fertility, is the average number of subwords per tokenized word. It requires a stated word-segmentation convention and is not the same denominator as a parallel-sentence token ratio. Neither metric directly measures downstream task quality. [127][123]

Formatting can be equally surprising. Published cl100k_base regression fixtures show today followed by newline-space taking three tokens, while adding one more newline makes the string two tokens. More input characters produced fewer tokens because the complete pattern admitted a different segmentation. Indentation, punctuation, escaping, and numeric grouping can similarly alter counts—but only text presented to the model matters. Extra syntax in an HTTP envelope does not count unless the service serializes it into model-visible content. [107][85][128]

Joining text changes boundaries

Tokenization is generally not additive under string concatenation. Encoding a + b can differ from concatenating encode(a) and encode(b) because the joined text changes pre-tokenization regions or creates adjacent units eligible for a merge. This matters for prompt prefixes, leading spaces, suffixes, partial words, and labels that look indivisible to a human. [96][107]

Joined encoding is not fragment addition

Both ID sequences can decode to the text 00, so decoding and re-encoding [15,15] need not preserve those original IDs. For counting and caching, always serialize the complete request before encoding rather than summing arbitrary fragments. [107]

Test behavior, not a token-count folk rule

Boundaries can also complicate character-sensitive work, but they are not a complete causal explanation. The 2024 CUTE study found that tested models could often spell tokens yet struggled more with character insertion, deletion, substitution, and swapping. The 2025 CharBench preprint found different relationships for counting and character-position tasks, with word length and requested counts often mattering more than token count alone. These results rule out both extremes: tokenized models are not categorically blind to spelling, and fewer tokens do not universally solve character tasks. [129][130]

A boundary change is a reason to run a controlled behavior test, not proof that a rewritten prompt is better. Prompting and In-Context Learning covers instructions, demonstrations, ordering, and held-out prompt experiments. [111]

Count the complete request

The relevant input count belongs to the complete representation presented to the model: system instructions, serialized message history, role and boundary markers, retrieved content, and model-visible tool or schema definitions. The HTTP JSON envelope is a transport representation and should not be tokenized as a substitute unless its literal syntax is itself included in model content. [131][132][128]

What normally belongs in model input

Exact local counting requires the intended tokenizer artifacts, chat-template revision, special-token settings, and complete joined input. Pinning a repository revision does not freeze caller overrides. Adding vocabulary entries also requires resizing compatible model embedding parameters; a tokenizer with attractive counts cannot simply replace the checkpoint’s tokenizer. [114][105][133]

Exact and estimated hosted counts

Hosted counters have provider-specific contracts. OpenAI documents its Responses input-count endpoint as returning the exact input count the model will receive, including role and boundary formatting. Claude documents its token-counting result as an estimate that can differ slightly from message-creation usage and may include automatically added material. Use the provider’s stated guarantee rather than generalizing one service’s semantics to another. Noah Hein’s AI Engineering 101 workshop illustrates the practical value of counting documents with the selected model tokenizer before processing them. [128][134][133]

Reserve room for generation

A context window is a bound on the token sequence available to one model invocation. For an endpoint with a shared context limit, let $P$ be the complete counted prompt, $R$ the generated-token allowance reserved by the application, and $C$ the supported context capacity. If the endpoint also has a separate maximum output $O$, both bounds must hold: [135][136][137]

A max_new_tokens-style setting caps generated tokens; it does not prove that prompt plus generation fits the shared context. Provider-defined hidden reasoning or formatting tokens may also consume the generated allowance, so reserve according to the actual endpoint contract without counting the same category twice. [119][128]

Different ways a bounded request can end

Cached prefixes still occupy context even when caching reduces processing charges. Likewise, fitting the nominal window does not guarantee that every retained fact influences the answer. Controlled long-context experiments have found position-dependent use in several tested models and tasks. Choosing what deserves the finite allowance is context allocation, covered in Context Engineering; architectural reach and effective use are discussed in When longer context helps. [137][60]

From token usage to charges

Providers bill according to named usage categories rather than one universal token rate. Read the pricing table and usage schema together because input, cached input, and generated output can have different prices and reporting rules. [128][140]

A dated accounting example

Cached is not absent

Claude reports cache reads, cache creation, and ordinary input separately. Add the documented categories when reconciling total input; cached prefixes still occupy context. [141][137]

Tokenization therefore affects both capacity and charges: language, formatting, templates, and tool definitions can change counted input. But price per token alone cannot compare models whose tokenizers, output lengths, and task results differ. Token counts also affect computational work without determining elapsed latency; concurrency, batching, hardware, request rate, and output behavior matter. AI Cost and Performance Engineering owns those workload comparisons, while Inference Engineering covers serving behavior and bottlenecks. [123][142]

Open questions

1. 01

   Reliable length transfer remains difficult because positional adaptation and task competence can change separately. Progress would preserve short-input capabilities while improving retrieval and multi-step use of longer inputs under controlled comparisons. [59][61]

2. 02

   Which sparse connections preserve necessary dependencies across workloads? Pair counts alone cannot establish usefulness. [55]

3. 03

   Faithful explanations must connect intermediate mixtures to downstream behavior despite interacting components. Progress would produce interpretations that predict controlled intervention outcomes on new inputs, with explicit limits on their scope. [62][64]

4. 04

   Can future byte- or patch-based model architectures retain open input coverage while matching subword systems’ practical training and serving efficiency? BLT concentrates expensive processing at uncertain byte regions, but operation counts do not yet establish equivalent wall-clock behavior or broad deployment. Progress would require controlled quality, latency, memory, and robustness comparisons at matched training resources. [143]

5. 05

   How should multilingual tokenizer quality be evaluated when word boundaries, scripts, morphology, and downstream tasks differ? Existing fertility and parallel-corpus ratios reveal representation disparities but do not measure competence. Progress would combine explicit denominators, versioned tokenizers, representative workloads, and task outcomes across languages. [127][123]

6. 06

   How much of character-level failure is caused by token boundaries rather than data, architecture, prompting, or task formulation? Recent studies find task-dependent relationships and show that models can possess spelling knowledge while failing at manipulation. Stronger causal evidence would hold the model and training data fixed while varying representation and evaluating multiple character operations. [129][130]

7. 07

   Can provider token accounting become portable without hiding model-specific serialization? Current APIs expose different guarantees and category semantics. Progress would look like versioned, machine-readable contracts that identify tokenizer artifacts, template revisions, model-visible additions, count guarantees, and billing-category relationships. [134][128][141]