Tokenization: How Text Becomes the Units Models Learn and Generate

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. Hugging Face: Tokenizers

   A tokenizer converts text into a sequence of vocabulary identifiers that a model can process numerically. Tokens can represent words, characters or subword fragments; one word need not equal one token. Subword tokenization balances vocabulary size against the ability to represent unfamiliar words. Encoding performs tokenization and conversion to IDs; decoding maps IDs back into text.

2. Tokenization algorithms — Transformers

   A tokenizer represents text using a finite vocabulary rather than treating each word as an indivisible unit. Subword methods can keep frequent words intact and split less common words into smaller pieces. BPE learns merge rules by repeatedly combining frequent adjacent units; byte-level BPE begins from byte values so it can encode arbitrary input bytes without an unknown-word token. The same text can split differently under different vocabularies. Tokenization therefore changes sequence length and the units available to the model; a token count is not a word count.

3. OpenAI Cookbook: How to count tokens with Tiktoken

   The published cl100k_base example encodes "2 + 2 = 4" as [17,489,220,17,284,220,19], corresponding to byte pieces ["2"," +"," ","2"," ="," ","4"]. Repeated occurrences reuse IDs, and spaces can belong to punctuation pieces or stand alone. The same string uses five tokens under r50k_base. For "お誕生日おめでとう", the published counts are 14 under r50k_base, nine under cl100k_base and eight under o200k_base. In cl100k_base, the bytes of 誕 are split between tokens 45918 and 243.

4. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

   A token ID identifies a vocabulary entry; its embedding supplies the numeric representation used to express relationships and perform model computation.

5. Attention Is All You Need

   Learned embeddings map discrete vocabulary tokens to numerical vectors. Positional information supplies ordering, and decoder attention masks future positions so predictions depend only on available preceding output. A learned linear projection and softmax convert decoder representations into next-token probabilities; for vocabulary scores z, softmax(z)j=exp(zj)/Σv exp(zv). These are computations using weights. Training instead updates weights with an optimizer; the paper uses Adam. Recomputing representations for a new context is consequently different from learning new parameters.

6. Unicode Glossary

   A Unicode code point is an integer position in the Unicode codespace; not every position has an assigned character. An encoded character associates an abstract character with a code point. A code unit is a unit of encoded text: UTF-8 uses eight-bit units, UTF-16 sixteen-bit units, and UTF-32 thirty-two-bit units. These are character-encoding units, distinct from a model tokenizer's vocabulary identifiers.

7. Unicode Standard Annex #29: Unicode Text Segmentation

   Grapheme clusters are a programmatically determined approximation to user-perceived characters. A perceived character can contain multiple code points: the annex illustrates a base letter followed by a combining accent. Extended grapheme clusters provide default boundaries, with permitted tailoring. Unicode word boundaries also require language-sensitive treatment; reliable word segmentation for languages including Thai, Chinese and Japanese can require dictionary lookup or other mechanisms.

8. Unicode FAQ: UTF-8, UTF-16, UTF-32 and BOM

   UTF-8 represents Unicode scalar values using sequences of one to four bytes. Valid multibyte sequences have constraints on their leading and continuation bytes; arbitrary byte sequences need not be valid UTF-8. A conforming decoder must handle malformed sequences as errors rather than interpret them as characters, potentially reporting an error or inserting U+FFFD, the replacement character.

9. Unicode normalization charts: Latin

   The normalization chart gives U+00E9, é, as its own NFC and NFKC result, while NFD and NFKD produce U+0065 U+0301. This supplies an exact composed-versus-decomposed accent example without assuming that a tokenizer performs either transformation.

10. Unicode 16.0 Core Specification, Chapter 3: encoding forms

    Unicode specifies UTF-16 surrogate pairs for scalar values above U+FFFF and the bit distributions for UTF-8. Applying these rules gives U+00E9 the UTF-8 bytes C3 A9, while U+0065 U+0301 becomes 65 CC 81. For the constructed sequence U+1F469 U+200D U+1F4BB, UTF-8 is F0 9F 91 A9 E2 80 8D F0 9F 92 BB. It occupies three code points, five UTF-16 code units and eleven UTF-8 bytes. In A👩‍💻B, the emoji therefore spans [1,4) in code-point coordinates, [1,6) in UTF-16 units and [1,12) in UTF-8 bytes.

11. Unicode Standard Annex #15: Unicode Normalization Forms

    NFD performs canonical decomposition; NFC additionally performs canonical composition. NFKD and NFKC also apply compatibility decompositions. Canonically equivalent representations include a precomposed accented letter and its base-plus-combining-mark sequence. Compatibility normalization can remove distinctions: the fi ligature becomes two letters, and superscript formatting can disappear. Normalization therefore supplies a consistent representation under a chosen equivalence relation, rather than preserving every original code-point sequence.

12. Hugging Face Tokenizers: The tokenization pipeline

    The library separates normalization, pre-tokenization, the segmentation model and post-processing. Normalization can change text; its NFD-plus-accent-removal example converts accented letters to unaccented ones. Pre-tokenization establishes regions within which the segmentation model operates, constraining possible final tokens. The model applies learned segmentation rules and maps pieces to vocabulary IDs; post-processing can insert special tokens. The documented Whitespace pre-tokenizer separates punctuation, including the apostrophe in I'm. Combining it with individual-digit splitting turns 911 into three separate digit regions. Returned spans refer to positions in the original sentence.

13. tiktoken 0.11.0: encoding definitions

    The cl100k_base and o200k_base definitions separate the pretokenization regular expression, mergeable ranks and special-token mapping. Their numeric expression matches runs of one to three numeric characters. Applying that expression to constructed ASCII examples gives regions 123|4 and 123|456|7; these are pretokenization regions, not asserted final token sequences. GPT-2's expression instead permits unrestricted numeric runs. The definitions also specify expected hashes for downloaded vocabulary artifacts, making the artifact identity more precise than the encoding name alone.

14. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    Whole-word tokenization trades a growing vocabulary for difficulty representing unknown or misspelled words.

15. Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code

    Subword tokenization can resemble meaningful word decomposition, but its frequency-driven splits need not match human linguistic intuition.

16. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

    ByT5 processes UTF-8 bytes using 256 byte values plus special IDs, removing the learned subword vocabulary and its preprocessing pipeline. This reduces vocabulary-related parameters but produces longer sequences, creating a different resource tradeoff. Against mT5, the study reports stronger results on tested spelling- and pronunciation-sensitive tasks and greater resilience to synthetic character noise. Inference-speed differences vary substantially by task; short word-level tasks behave differently from sentence classification and document summarization. Byte-based representation therefore remains a practical alternative rather than an obsolete precursor to subwords.

17. SentencePiece configuration options

    SentencePiece supports Unigram, BPE, word and character models, so its name does not identify one segmentation algorithm. Its byte_fallback option represents otherwise out-of-vocabulary characters with UTF-8 byte tokens; this differs from starting every input region with bytes. Without fallback, excluded characters can become an unknown token whose decoded surface is a configured placeholder. Normalization and whitespace handling are separately configurable, including compatibility normalization, dummy prefixes and removal of extra whitespace. Control symbols are not recognized from raw text and decode to empty strings, whereas user-defined symbols are matched from raw text.

18. Training an LLM from Scratch, Locally

    Character-level tokenization keeps the workshop vocabulary small but requires more tokens and more composition to represent meaningful text.

19. Training an LLM from Scratch, Locally

    A large vocabulary can make the embedding table disproportionately large for a tiny model.

20. Philip Gage: A New Algorithm for Data Compression

    Gage's February 1994 article presents BPE as general-purpose data compression with a small, fast decompression routine suited to limited-memory applications. It repeatedly replaces frequent adjacent byte pairs with unused byte values and stores the substitution table alongside the compressed block. Its example transforms ABABCABCD into XXCXCD by replacing AB, then into XYYD by replacing XC. Compression works on buffered blocks with separate substitution tables; slow compression is an acknowledged tradeoff.

21. Excellent Papers for 2012 — Japanese and Korean Voice Search

    Google's account of Schuster and Nakajima's 2012 speech-recognition work identifies two related problems: representing an open-ended vocabulary without unrecognized words, and handling text without explicit spaces between words. It describes statistical segmentation as part of the solution used in Google's Japanese and Korean Android voice-search systems, alongside language modeling and dictionary construction.

22. Neural Machine Translation of Rare Words with Subword Units

    The 2016 ACL paper adapts BPE to handle rare-word translation within the neural model instead of relying on dictionary fallback. Names, compounds and changing word forms motivate smaller reusable units: copying an unknown word cannot supply every required transliteration or grammatical change.

23. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

    Kudo and Richardson's 2018 system paper addresses dependence on language-specific preprocessing, including separate word segmenters and hand-written detokenization rules. SentencePiece supports training BPE or Unigram segmentation directly from raw sentences. Its model artifact includes vocabulary entries, segmentation parameters and compiled normalization rules, addressing reproducibility problems caused by separately configured preprocessing tools. The paper's reconstruction contract preserves normalized text; it does not promise recovery of distinctions removed before segmentation.

24. tiktoken educational byte-pair encoding implementation

    The educational encoder splits text with a configured regular expression, converts each region to UTF-8 bytes and begins with individual-byte pieces. It repeatedly merges the eligible adjacent pair with the smallest stored rank, then looks up the final pieces' IDs. Encoding uses fixed ranks; its separate trainer counts corpus pairs and updates the segmented training data after each merge. An illustrative consequence is that ranks bc before ab segment abc as a|bc when no further merge exists, whereas left-to-right longest matching chooses ab|c. Ranked merging therefore need not equal longest vocabulary matching.

25. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    Byte-pair encoding (BPE) repeatedly merges frequent adjacent token pairs, growing the vocabulary while shortening the corpus representation.

26. Neural Machine Translation of Rare Words with Subword Units

    The paper adapts byte-pair compression to character-sequence segmentation. It initializes words as characters plus an end-of-word marker, counts adjacent pairs weighted by word frequency, merges the most frequent pair and repeats. Algorithm 1 uses low:5, lower:2, newest:6 and widest:3. In that initial corpus, e–s and s–t each have weighted count nine, illustrating why a reproducible walkthrough must specify tie handling. This implementation excludes pairs crossing word boundaries. Learned operations are then applied to new words rather than relearned from each input.

27. How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

    The demonstrated GPT-2 tokenizer uses the trained merge vocabulary and rankings to turn new text into the model's existing tokens.

28. BERT original WordPiece tokenizer implementation

    BERT's WordpieceTokenizer chooses the longest vocabulary match from the current position, then advances and repeats. Noninitial pieces use the continuation prefix ##. Its documented example segments unaffable into un, ##aff and ##able. If any remaining span cannot be matched, the implementation emits the unknown token for the entire word rather than retaining the partial segmentation. It also maps words beyond its configured character limit to the unknown token.

29. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

    The proposed Unigram tokenizer assigns each vocabulary piece a probability and scores a segmentation by multiplying its pieces' probabilities. The best segmentation maximizes that score over candidate segmentations and can be found with the Viterbi algorithm. The same vocabulary can therefore admit several segmentations with different scores; the paper also describes sampling alternatives. This differs from choosing merges by BPE rank or greedily selecting the longest next piece.

30. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

    Taku Kudo’s July 2018 ACL paper treats alternative segmentations as useful training variation. Sampling them improves translation robustness, particularly in lower-resource and out-of-domain experiments. Without sampling, Unigram and BPE achieve broadly comparable translation scores; the benefit is therefore not simply replacing BPE with another deterministic encoder.

31. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

    Yonghui Wu and colleagues’ 2016 GNMT report explicitly adopts WordPiece from Google’s Japanese/Korean speech-recognition work to address rare-word translation. Shared source and target segmentation helps the model learn copying, while word-boundary markers support reconstruction. Its implementation still maps excluded rare characters to an unknown character. The experiments compare word, character, mixed and 8K–32K WordPiece vocabularies. On WMT English–French, the character model achieves translation quality close to WordPiece but takes 1.0530 CPU seconds per sentence versus 0.2118 for WPM-32K, illustrating the cost of longer sequences.

32. A Neural Network for Machine Translation, at Production Scale

    On September 27, 2016, Google Brain researchers Quoc Le and Mike Schuster announced GNMT and stated that Google Translate’s mobile and web applications were already using it for all Chinese-to-English machine translations. They identified handling rare words through smaller units as one contributing development, while acknowledging remaining dropped words and mistranslated names or rare terms.

33. Transformers Tokenizer API

    Transformers exposes decoding options that remove special tokens and clean tokenization spaces, so decoded output depends on more than the ID sequence. Adding vocabulary entries assigns new IDs and requires resizing the model's token embedding matrix to match the tokenizer. Added tokens are treated differently from entries governed by the underlying segmentation algorithm. apply_chat_template returns the conversation's token IDs, including control tokens, and accepts tool definitions as additional template inputs.

34. tiktoken Encoding API implementation

    tiktoken distinguishes ordinary text encoding from explicitly allowed special-token recognition. By default, encode rejects recognized special-token spellings; disallowed_special=() treats them as ordinary text, while allowed_special permits reserved IDs. decode first reconstructs bytes and then decodes UTF-8, replacing invalid sequences by default; decode_single_token_bytes preserves a token's bytes. Independently decoding fragments can therefore lose information. decode_with_offsets maps a token beginning inside a multibyte character to that character's index. encode_with_unstable exposes stable prefix tokens and possible completions, demonstrating that an unfinished suffix need not retain its segmentation.

35. tiktoken encoding regression tests

    The tests specify GPT-2 encoding of hello world as [31373,995] and cl100k_base encoding as [15339,1917]. GPT-2 encodes 0 as [15] but 00 as [405]: joining two separately encoded zeros changes both segmentation and count. Consequently, decoding [15,15] to 00 and re-encoding need not preserve the original IDs. For cl100k_base, today followed by newline-space has three tokens, while adding another newline produces two. The thumbs-up emoji is three tokens in its fixture. The suite also checks whitespace and multilingual text round trips and explicit special-token policies.

36. Python codecs: Incremental Encoding and Decoding

    An incremental decoder retains state across calls, including undecoded input, so concatenating its outputs matches decoding the concatenated input. The final call must set final=True, which flushes buffered data and invokes error handling if an incomplete byte sequence remains. Strict handling raises an error; replacement handling inserts U+FFFD for decoding errors. This supplies a standard implementation mechanism for receiving UTF-8 bytes in fragments without treating every fragment as a complete string.

37. Claude Platform: Streaming messages

    Claude's streaming interface delivers named server-sent events containing JSON. A text_delta carries a text string, illustrated by the fragment ello frien, rather than a token ID. Content-block events, message updates, pings and errors have distinct roles. Usage counts in message_delta are cumulative. Consequently, applications should accumulate text deltas and read usage accounting rather than count delivery events as model tokens.

38. Utilities for Generation: TextStreamer and TextIteratorStreamer

    Transformers streamers receive generated token IDs and decode them for delivery. TextStreamer buffers until text forms complete words; end() flushes remaining cached text. TextIteratorStreamer instead puts printable text in a queue for a downstream consumer, with a configurable queue timeout for separately threaded generation. Thus a generation step, decoded text fragment and consumer observation are distinct events; one delivered fragment need not equal one token. Incremental delivery can expose a prefix before generation finishes.

39. Language Models are Unsupervised Multitask Learners

    Tokenization represents text as a sequence of vocabulary symbols rather than necessarily whole words. GPT-2 uses byte-level byte-pair encoding: a 256-byte base supports arbitrary text, while frequency-based merges create longer tokens, subject to character-category restrictions. Autoregressive modeling factorizes sequence probability as P(x1,…,xn)=∏i Pθ(xi|x<i). Generation repeatedly predicts a next-token distribution conditioned on the supplied prefix and previously generated tokens. Task instructions and examples can therefore change the conditional distribution without changing θ. GPT-2's zero-shot evaluations modify neither learned parameters nor architecture; request text changes the inputs and resulting activations, not the trained weights.

40. Transformers: greedy and sampled token selection

    Greedy decoding chooses the highest-probability next token at each step. Multinomial sampling draws a token from the vocabulary probability distribution, so lower-probability tokens can be selected. Transformers enables ordinary sampling with do_sample=True and num_beams=1. Combined with the transformer's autoregressive architecture, the selected token becomes part of the input for the next prediction. Greedy selection removes this token-draw randomness but does not establish semantic correctness or globally maximize the probability of an entire sequence.

41. Hugging Face Tokenizers: NormalizedString implementation and alignment tests

    NormalizedString retains original and normalized strings with an alignment entry for each normalized byte, and distinguishes Original from Normalized ranges. Its NFKC slicing test starts with mathematical double-struck text 𝔾𝕠𝕠𝕕 𝕞𝕠𝕣𝕟𝕚𝕟𝕘. Original range [0,4) retrieves original 𝔾 and normalized G; normalized range [0,4) retrieves normalized Good and original 𝔾𝕠𝕠𝕕. These are UTF-8 byte coordinates: the same numerical range selects different spans depending on its coordinate system.

42. Transformers: Utilities for Tokenizers

    from_pretrained accepts a revision identifying a branch, tag or commit, and initialization arguments can override special-token settings. Pinning repository artifacts therefore does not by itself freeze caller-supplied configuration. The API documents return_offsets_mapping as character start/end pairs available through fast tokenizers. It separately warns that encoding individual chat messages may be imperfect when a template's conversation prefix cannot be distinguished from message-start material.

43. Transformers: model-specific chat serialization

    A chat template converts ordered role/content messages into the token sequence a causal model continues, inserting model-specific role markers, message boundaries and special tokens. Models fine-tuned from the same base can require different formats: Mistral-Instruct brackets user messages with instruction delimiters, while Zephyr uses explicit speaker markers. The guide warns that incompatible control tokens degrade performance. Where the format requires it, add_generation_prompt=True appends the assistant-start marker; omitting it can cause continuation of the user's message instead of a reply. Some templates need no such marker. Templates already supply required special tokens: use apply_chat_template(tokenize=True), or tokenize the rendered string with add_special_tokens=False to avoid duplicated beginning/end tokens.

44. Chat templates — Transformers

    Chat roles and message contents are serialized into a token sequence using a model-specific template. Control tokens can mark message boundaries and speaker roles; even models derived from the same base can require different formats. A generation prompt marks the start of a new assistant response, while continuing a final message serves a different purpose. Templates already include necessary special tokens, so adding them again during a second tokenization step can duplicate boundaries and harm behavior. A chat API abstraction does not exempt the input from the underlying sequence format.

45. Mistral-7B-Instruct-v0.1 tokenizer configuration at b54db9339734a22dd2b55531bd8146540729fbb8

    This pinned configuration assigns BOS <s> ID 1 and EOS </s> ID 2, enables automatic BOS insertion and disables automatic EOS insertion. Its template separately emits BOS, brackets user content with instruction delimiters and appends EOS after assistant content. Applying the inspected template by hand to user Hi, assistant Hello, user Bye gives <s> [INST] Hi [/INST] Hello</s> [INST] Bye [/INST]. This template does not reference add_generation_prompt, so that option adds no separate assistant prefix.

46. Llama 3.3 Prompt Formats

    Llama's documented format distinguishes prompt start, role-header boundaries, padding and ending markers. The guide describes end_of_text as a base-model stopping marker, eot_id as completion of an assistant turn and eom_id as a handoff point during interaction. Its instruct example serializes system and user messages with headers and endings, then leaves an assistant header ready for continuation. Padding makes sequences the same length within a batch.

47. Transformers GenerationConfig: output reservation and termination

    Generation selects tokens from successive vocabulary scores using the decoding configuration. max_new_tokens caps generated tokens without counting prompt tokens. Consequently, setting it to R reserves output length but does not itself establish that prompt length P plus R fits the model's context. eos_token_id identifies one or more end-of-sequence tokens, allowing generation to stop before the cap; minimum-length and other stopping controls can alter termination. stop_strings supplies additional textual stopping conditions. Reaching the output cap bounds the generation loop but does not establish that the response is complete. Application inference: validate the combined prompt/output budget before generation rather than treating an output-only limit as a context check.

48. Fixing bugs in Gemma, Llama & Phi-3

    Check that training and inference insert the same single beginning-of-sentence (BOS) token where the model requires one.

49. Fixing bugs in Gemma, Llama & Phi-3

    Use distinct padding and end-of-sequence (EOS) token IDs when the training loss masks tokens by the padding ID.

50. SFT Trainer: objective, shifting, and masking

    For prompt x and target response y, supervised fine-tuning minimizes negative conditional log likelihood: L = -sum_t m_t log pi_theta(y_t | x,y_<t), optionally normalized by the count of included tokens. The mask m_t excludes padding and can exclude prompt or non-assistant tokens. Each prediction conditions on the demonstrated prefix, rather than a prefix sampled from the model; this is teacher forcing. The loss rewards probability assigned to the demonstrated next token, not execution success or truth directly. Current TRL supports completion-only and assistant-only loss with dataset/template requirements. Changing masking changes which behavior receives direct supervision, even when the visible example text is identical.

51. Language Model Tokenizers Introduce Unfairness Between Languages

    The study compares tokenized lengths of parallel sentences from FLORES-200, using 2,000 sentences translated into 200 languages. It defines tokenizer parity through the ratio of token counts for corresponding translations and observes substantial differences across the evaluated tokenizers and languages. For tokenizers with unknown symbols, it excludes language results exceeding its unknown-character threshold. The authors acknowledge that topic coverage, English-centric named entities and alternative translations can affect the measured ratios.

52. Training an LLM from Scratch, Locally

    Byte pair encoding (BPE) builds reusable tokens from recurring character patterns in the training data.

53. Training an LLM from Scratch, Locally

    An uncommon variable name need not have its own vocabulary entry; it can split into smaller pieces, increasing inference work.

54. Language Model Tokenizers Introduce Unfairness Between Languages

    Table 1 reports cl100k_base token-count premiums relative to English of 1.60 for French, 1.58 for German and 2.30 for Japanese on FLORES-200. These provide representative numerical examples of equivalent translated content occupying different token budgets.

55. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

    The paper defines subword fertility as the average number of subwords produced per tokenized word. Its separate continued-word metric measures how often words split into multiple pieces. The analysis uses training and development portions of specified Universal Dependencies v2.6 treebanks and finds different fragmentation patterns across languages and monolingual versus multilingual tokenizers. Fertility measures segmentation granularity under that word denominator; it is distinct from the paper's downstream task evaluations.

56. OpenAI API: Counting tokens

    OpenAI documents POST /v1/responses/input_tokens as accepting Responses-format inputs and returning the exact input count the model will receive, including role and boundary formatting absent from locally tokenized content. Tools and schemas also contribute. This is a provider-specific counting contract, distinct from the supplied Claude documentation's estimate contract. The guide additionally states that output usage can include invisible channel, tool-call and message-formatting tokens even when reasoning_tokens is zero. Output-token limits cover those generated tokens as well as visible text; the difference is not a fixed overhead.

57. CUTE: Measuring LLMs’ Understanding of Their Tokens

    Lukas Edman, Helmut Schmid and Alexander Fraser’s November 2024 CUTE paper tests orthographic knowledge—the spelling and character composition of text. With four demonstration examples, evaluated models perform well at spelling tokens but struggle more with character insertion, deletion, substitution and swapping than with corresponding word operations. Knowing a token’s spelling therefore does not establish reliable character manipulation, while successful spelling contradicts a categorical claim that subword models cannot access character information.

58. CharBench: Evaluating the Role of Tokenization in Character-Level Tasks

    Omri Uzan and Yuval Pinter’s August 4, 2025 preprint distinguishes character counting from locating characters within words. Across its evaluated models, token count and average characters per token correlate relatively weakly with correctness. Word length and the required count matter more for counting tasks. For positional tasks, longer tokens containing the queried character correlate with lower accuracy, particularly for GPT-4o. Tokenization’s relationship to errors consequently depends on the task rather than following a universal fewer-tokens-is-better rule.

59. Transformers: chat templates serialize conversation history

    A chat model still receives a token sequence. A chat template turns role-labelled messages into that sequence, adding model-specific control tokens that mark speakers and message boundaries. Retained user messages, assistant messages and instructions therefore consume input tokens alongside the latest question. The rendered template, including any assistant-generation prefix, determines the effective input. Counting only visible message text can miss control-token overhead. Stored conversation history is not automatically identical to the history actually serialized for a particular inference call.

60. Understanding and counting tokens — OpenAI

    Tokens are the units a model processes: a token can represent a character, word fragment, whole word or punctuation, and spaces affect segmentation. Counts depend on the model encoding, language and surrounding text; token counts are not word counts. Counting only plain text can omit message formatting, tool definitions, schemas and non-text inputs. Reasoning models can generate internal reasoning tokens that are not visible answer text but still count toward output usage. An input count does not predict output length.

61. AI Engineering 101

    The workshop treats tokenization as model-coupled and counts encoded tokens per document before processing.

62. Claude Platform: Token counting

    Claude's counting endpoint accepts structured message inputs, including system prompts and tool definitions, rather than only message-content strings. The documentation explicitly describes the returned input count as an estimate that can differ slightly from the input count used when creating a message. Counts may include automatically added system material. This provides a concrete distinction between counting a specified representation and guaranteeing an eventual provider request count.

63. Transformers LlamaConfig: supported sequence positions

    LlamaConfig defines max_position_embeddings as the maximum sequence length for the model and separately documents RoPE parameters for positional scaling. Application budgeting inference: let C be the supported limit for the exact checkpoint and positional configuration, P the length of the complete tokenized prompt, and R the reserved generated-token count; enforce P+R<=C. P includes template/control tokens, system instructions, conversation and retrieved text. Count the final serialized token IDs rather than characters or independently tokenized fragments. If the budget fails, remove or summarize selected material, reduce retrieval or reserve fewer output tokens, then serialize and count again.

64. Conversation state: managing the context window — OpenAI

    The context window limits tokens used in one request, including supplied input and generated output; applicable reasoning tokens also consume capacity. Instructions, conversation history, retrieved material and tool results supplied to that invocation therefore share its input budget. A simple worked design reserves generated-token capacity before allocating remaining space to input, while also respecting the model's separate output limit. For a reasoning model, reserve space for hidden reasoning as well as the visible answer without counting the same output tokens twice. Persisting a conversation does not make its usable context unbounded.

65. Claude context windows and cached-token accounting

    Cached prefixes still occupy the context window. System instructions, tool definitions, messages and tool results consume capacity, and generated output also occupies the window. A useful planning constraint is I + O <= W, where I is counted input, O is generated output and W is the model's context capacity. Caching changes processing cost, not this capacity requirement. Inputs exceeding the window are rejected; generation at the limit follows model-specific overflow behavior.

66. Padding and truncation — Transformers

    Tokenizer truncation limits the input representation before model execution. With truncation enabled, max_length specifies the retained length or defaults to the model maximum when available. For paired sequences, longest_first repeatedly removes tokens from the longer sequence; only_first and only_second select the sequence eligible for truncation. No truncation is the documented default. Removing input tokens discards evidence; it is different from limiting newly generated output tokens. A tokenizer setting does not determine whether a remote API rejects or truncates an oversized request.

67. Claude Platform: Stop reasons and fallback

    Claude distinguishes natural completion through end_turn from reaching the requested output cap through max_tokens and filling the context window through model_context_window_exceeded. These reasons can accompany successful API responses, so HTTP success does not establish that the answer is complete. During streaming, stop_reason begins as null and is supplied through message_delta.

68. Transformers GenerationConfig: stopping and output budgets

    Generation can end on an end-of-sequence token, a configured stop string, a length limit, or other stopping criteria. max_new_tokens limits generated tokens independently of prompt length; max_length includes the prompt. These are upper bounds, not promises to produce that many tokens. max_time is not a hard deadline: generation finishes its current pass after the allotted time. For ordinary single-beam generation, do_sample selects sampling versus greedy decoding. An application should distinguish normal completion from length truncation, cancellation and failure rather than interpreting every returned prefix as a completed answer.

69. Lost in the Middle: How Language Models Use Long Contexts

    The paper tests multi-document question answering and synthetic key-value retrieval while controlling context length and where relevant information appears. Reordering the answer-containing document changes its position without changing the desired answer. Several evaluated models performed better with evidence near the beginning or end than in the middle; results varied by model and task. This separates fitting tokens inside a supported context window from reliably retrieving and using their information. It motivates controlled position and distractor tests rather than assuming nominal length measures effective use.

70. Claude Platform: Pricing

    The inspected table prices Claude Sonnet 4.6 standard input at $3 and output at $15 per million tokens; cache reads cost $0.30, five-minute writes $3.75 and one-hour writes $6 per million. A constructed request with 1,000 ordinary input tokens and 200 output tokens therefore costs $0.006 before other charges. The tool-use table specifies 497 additional system-prompt tokens for this model with auto tool choice and at least one tool. At the ordinary input rate, that overhead alone adds $0.001491, excluding tool definitions and results.

71. Claude Platform: Prompt caching

    Claude reports cache reads, cache creation and ordinary input as separate input categories. Its input_tokens field excludes tokens read from or written to the cache. Total input is cache_read_input_tokens plus cache_creation_input_tokens plus input_tokens. The documentation's example has 100,000 cache-read tokens, no cache creation and 50 ordinary input tokens, totaling 100,050 input tokens. Reading only input_tokens would therefore substantially understate the request representation.

72. GenAI-Perf: latency and throughput measurement

    GenAI-Perf defines TTFT as first-response receipt minus request-send time, and request latency as final-response receipt minus send time. Its inter-token latency divides the interval between successive responses by the latter response's generated-token count. Output-token throughput is total generated tokens divided by benchmark duration; request throughput is final responses divided by duration. A p99 latency is the 99th percentile of the specified latency observations, not an average. The tool supports concurrency, request rate, warmup count, measurement intervals, and input/output-length controls. Reproducible comparisons must report those settings and the workload/tokenizer.

73. Byte Latent Transformer: Patches Scale Better Than Tokens

    Artidoro Pagnoni and colleagues at FAIR introduced BLT in a December 13, 2024 preprint. It addresses the long sequences produced by byte-level modeling by grouping bytes into variable-length patches without a fixed patch vocabulary. A small byte model estimates next-byte uncertainty to choose boundaries, concentrating expensive processing where prediction is harder. Experiments extending to eight billion parameters demonstrate feasible scaling and improved results on tested noisy-input and character tasks relative to the paper’s token-based baselines.

74. Unicode Emoji 15.1: emoji ZWJ sequences

    Unicode's versioned emoji data identifies 👩‍💻, woman technologist, as the sequence U+1F469 U+200D U+1F4BB. It provides a concrete three-code-point emoji for distinguishing displayed text from its underlying sequence.

75. MCP = Mega Context Problem - Matt Carey

    Creating and loading a separate tool for every API endpoint can make the tool descriptions themselves exceed a practical context budget.