Pretraining and Midtraining

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. Training Compute-Optimal Large Language Models

   Chinchilla estimates model-size and training-token allocation using training-curve envelopes, equal-compute comparisons and a fitted loss function. Its results favor approximately equal proportional scaling of parameters and tokens as compute grows. Unlike the earlier Kaplan analysis, it varies training horizons and matches learning-rate decay to those horizons, identifying schedule mismatch as one contributor to differing conclusions. Its simplified dense-transformer compute relationship is C approximately 6ND, where N counts parameters and D counts processed training tokens. The scaling analysis assumes training exposure remains below the available corpus size.

2. Reconciling Kaplan and Chinchilla Scaling Laws

   Pearce and Song investigate another explanation for the differing Kaplan and Chinchilla prescriptions: excluding embedding parameters and their computation matters substantially when fitting small models. Their analytic reconstruction, simulated training curves and small-model experiments recover different allocation exponents simply by changing this accounting. In their experiments, changing token horizons and decay schemes has a smaller effect. Parameter definitions and fitting range therefore belong alongside schedules when interpreting scaling-law disagreements.

3. Language Models are Few-Shot Learners

   Autoregressive pretraining learns to predict continuations from a broad mixture of text. Task-specific fine-tuning instead updates weights on examples directed at a target task; in-context examples condition inference without gradient updates. The likelihood objective can remain next-token prediction while the data distribution changes.

4. Optimizing LLMs for Speed and Memory

   Model parameters are numerical weight matrices and vectors loaded from a checkpoint; text inputs are represented separately as sequences of vectors. In ordinary inference, request inputs pass through those weights without a training update. Applied to RAG, instructions, conversation history, the question and retrieved text belong to request context when included in the input. Changing that text changes the computation without rewriting the checkpoint. Request-specific cached attention keys and values are intermediate computation state, not newly learned model parameters.

5. Krea 2 Technical Report

   Krea’s June 2026 image-model report places a stage called midtraining after progressively higher-resolution pretraining and before supervised fine-tuning. The team describes its role as moving the model distribution toward downstream use and as the last stage where it typically adds capabilities such as high-resolution generation, domain coverage and text rendering. It separately describes SFT as biasing a small curated set toward aesthetic qualities, preference optimization as learning from preferred and rejected generations, and reinforcement learning as optimizing multiple rewards.

6. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

   GLM-4.5's report distinguishes broad pretraining from subsequent domain-focused stages that it calls mid-training, explicitly including instruction data. Its corpus includes web documents, books, papers, code and multilingual text. Later pretraining upweights code, mathematics and science. Repository-level midtraining concatenates related files, issues, pull requests and commits, while increasing sequence length from 4K to 32K. Further stages incorporate synthetic reasoning and agent trajectories and extend sequences to 128K. The report also changes truncation to best-fit packing for midtraining to preserve reasoning and repository examples.

7. Training language models to follow instructions with human feedback

   The training loop separates demonstrations, ranked responses, and fresh policy rollouts. A scalar reward model learns preferences via -log sigmoid(r(x,y_w)-r(x,y_l)); the absolute score is less meaningful than the difference for the same prompt. PPO then generates responses and optimizes their learned reward with a per-token penalty for departure from the supervised reference. In expectation the log-probability-ratio penalty gives KL(pi_theta || pi_ref). A learned value function estimates expected return for advantage estimation. The paper also mixes pretraining gradients into some RL runs to reduce regressions on retained NLP tasks. It partitions prompts by user identity, rather than randomly splitting near-related prompts across sets.

8. From token identifiers to next-token probabilities

   Learned embeddings map discrete token identifiers to numerical vectors; positional information supplies sequence order. Transformer attention and feed-forward layers transform these representations. A learned output projection followed by softmax converts decoder outputs into a probability distribution over the vocabulary. Causal decoder masking prevents a position from using future tokens, allowing training against known next-token targets without revealing their answers. At generation time, the decoder consumes previously generated symbols when producing the next one.

9. Megatron-LM GPT dataset implementation

   Megatron's left-to-right mask builder returns separate attention masks, loss masks and position IDs. With attention resetting enabled, positions after an end-of-document boundary cannot attend to positions through that boundary. Independently, eod_mask_loss sets loss-mask entries at end-of-document input positions to zero. Position resetting is another independent option. Thus placing documents in one sequence, restricting cross-document visibility and excluding prediction positions from loss are separate configuration decisions.

10. PyTorch CrossEntropyLoss

    For integer targets, CrossEntropyLoss can return individual losses, sum them, or compute a weighted mean; ignored targets contribute neither loss nor the mean's denominator. With no class weights, averaging all valid token losses gives each scored token equal weight. Constructed example: documents with 100 and 900 scored tokens contribute 10% and 90% of the terms in a pooled token mean; averaging their two document means assigns each document 50%. These are different objectives despite identical documents.

11. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

    DoReMi defines domain weights as probabilities over datasets, with examples sampled uniformly within each selected domain. It distinguishes raw-token-count reference weights from optimized sampling weights. A small proxy learns weights by rescaling domain losses; averaged weights then determine the larger model's sampling distribution. Its experiments find that downweighting a domain can nevertheless improve that domain's perplexity. Constructed exposure example: selecting equally often between domains whose documents have 100 and 900 tokens yields approximately 10% and 90% processed-token shares when whole documents are consumed without truncation.

12. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    BERT constructs prediction targets from ordinary text: selected token positions are corrupted, and training predicts the original tokens using cross-entropy. The original text supplies the answers without separate human annotation for each prediction. This is a concrete example of self-supervised target construction, distinct from grouping observations into clusters. The paper subsequently fine-tunes pretrained parameters for downstream tasks.

13. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    T5's denoising construction replaces consecutive corrupted tokens with unique sentinel markers. The target contains the removed spans separated by corresponding sentinels and a final sentinel; it does not reproduce every uncorrupted input token. The baseline packs multiple sequences into batch entries and trains using target-sequence cross-entropy. The study changes individual experimental factors while holding others fixed, explicitly acknowledging that this can miss interactions. It uses validation results for most comparisons and checkpoint selection rather than repeatedly selecting against test results.

14. Efficient Training of Language Models to Fill in the Middle

    Fill-in-the-middle training rearranges a document's prefix, middle and suffix so the middle is predicted after both surrounding pieces, separated by sentinel tokens. This teaches infilling without replacing the causal decoder architecture. The study compares mixtures of transformed and ordinary text, assessing both left-to-right behavior and infilling. Some training choices barely change prediction loss yet substantially change executable code-infill results. Adding infilling to an already trained model also requires substantial additional training in the reported experiments, whereas including it during initial training preserves measured left-to-right performance.

15. TruthfulQA: Measuring How Models Mimic Human Falsehoods

    TruthfulQA distinguishes errors caused by imperfect learning from imitative falsehoods: false answers that are plausible under the text distribution a model learns to imitate. Its questions deliberately probe common misconceptions, and the tested models produced answers reflecting those misconceptions. Thus successful imitation of human text does not itself establish factual correctness.

16. A Neural Probabilistic Language Model

    Bengio and colleagues jointly learned word representations and next-word probabilities to address sparse observations: most possible word sequences never appear in training. Sharing learned representations lets observations inform predictions for related, unseen sequences. The paper compares this approach with n-gram models, which estimate probabilities from short word histories. This supplies an early motivation for learning reusable structure through language prediction.

17. A Fast Learning Algorithm for Deep Belief Nets

    Hinton, Osindero and Teh addressed the difficulty of learning densely connected networks with multiple hidden layers through greedy, layer-by-layer learning. The resulting parameters initialized a subsequent procedure that refined the whole generative model. Their demonstration modeled handwritten digit images and labels. Here, preliminary learning made a difficult network trainable; it was not broad language training intended to supply a reusable conversational model.

18. Deep Contextualized Word Representations

    ELMo learns contextual representations through forward and backward language modeling. For supervised downstream learning, it freezes the language-model weights and supplies a learned mixture of internal representations to a task-specific model. This offered a different reuse strategy from updating the entire pretrained model. The paper also reports domain-specific language-model adaptation before freezing, so feature reuse and continued pretraining can coexist.

19. Universal Language Model Fine-tuning for Text Classification

    ULMFiT separates general-domain language-model training, further language-model training on target-task text, and supervised classifier training. The expensive initial model can be reused across classification tasks instead of training each system from scratch. Its intermediate stage addresses differences between general text and the target distribution. The method uses an LSTM and techniques intended to limit overfitting and forgetting during adaptation, establishing that whole-model language transfer and intermediate domain adjustment did not depend on transformers.

20. Improving language understanding with unsupervised learning

    OpenAI's June 11, 2018 announcement describes training a transformer with a language-modeling signal, then adapting the same core model on smaller supervised datasets. The reported applications extend beyond document classification to reading comprehension, semantic similarity and commonsense tasks. The authors explicitly present the approach as combining existing transformer and unsupervised-pretraining ideas, and relate it to earlier sequence learning, ULMFiT and ELMo.

21. Scaling Laws for Neural Language Models

    Kaplan and colleagues fit empirical relationships between autoregressive language-model cross-entropy, non-embedding parameter count, dataset size and training compute. Their compute-efficient allocation favors larger models stopped well before convergence, with optimal parameter count scaling approximately as compute to the 0.73 power and data requirements as compute to the 0.27 power. The paper distinguishes parameter-limited, data-limited and compute-limited regimes and identifies tokenization-dependent constants.

22. 2 OLMo 2 Furious

    OLMo 2 calls its later capability-focused base-model stage mid-training. Initial training uses learning-rate warmup and cosine decay; the later stage changes to curated web, reference, instruction and synthetic math material while linearly reducing the learning rate to zero. Short microannealing runs test candidate math sources mixed with general web text before larger runs. The authors monitor math development performance alongside MMLU retention, restricting mixture-selection feedback to 200 GSM8K evaluation questions. Final models average separately trained branches with different data orders.

23. Unsupervised Cross-lingual Representation Learning at Scale

    XLM-R samples masked-language-model batches across 100 languages using a smoothed corpus-size distribution with alpha 0.3. Its ablations show that increasing alpha favors high-resource languages, while stronger smoothing gives low-resource languages more exposure. With fixed model capacity, adding languages initially helps low-resource transfer but eventually reduces performance through capacity dilution. The authors also report that downstream task performance continued improving after validation perplexity plateaued, so prediction loss alone was an insufficient stopping rule for their capability goal.

24. Curriculum Learning

    Bengio and colleagues formalize a curriculum as a sequence of training distributions produced by changing example weights. Their construction initially favors easier examples and progressively approaches the target distribution. A language experiment gradually admits windows containing less frequent vocabulary, comparing against training with the full vocabulary throughout. The paper reports benefits in its experiments while explicitly leaving effective definitions of difficulty and curriculum pace task-dependent.

25. Effective Long-Context Scaling of Foundation Models

    The authors continue Llama 2 training for 400 billion tokens, modifying positional encoding and using 32,768-token sequences for 7B/13B models and 16,384 for larger variants. They assess prediction loss, context probes and downstream tasks separately. Data ablations find that increasing the proportion of long documents alone does not consistently improve results. A separate 7B curriculum comparison holds total training tokens and tokens per update constant while changing when sequences grow from 4K to 32K; starting shorter reduces computation, with task-dependent differences in final scores.

26. The Llama 3 Herd of Models

    Meta describes Llama 3 pretraining as next-token prediction over a multilingual corpus, followed within its pretraining recipe by continued long-context training and a final annealing phase. Its reported mixture was roughly 50% general knowledge, 25% mathematics and reasoning, 17% code and 8% multilingual tokens, selected through smaller scaling experiments and larger-model evaluations. The mixture was not stationary: later changes increased non-English, mathematics and recent web data and reduced sources judged lower quality. Long-context continuation gradually increased the window from 8K to 128K and required both recovery on short-context evaluations and success on long-context probes.

27. Simple and Scalable Strategies to Continually Pre-train Large Language Models

    The authors separately track validation loss on earlier and incoming data while varying learning-rate rewarming, decay and previous-data replay. Rewarming can accelerate adaptation while increasing forgetting; replay comparisons keep continuation compute constant. The experiments cover English-to-English and English-to-German shifts at 405M parameters and a weaker shift at 10B. Selected combinations approach retraining-from-scratch baselines in average benchmark performance while requiring less incremental compute. Optimizer resets are intentional in the studied continuation setup; immediate large updates can produce transient loss spikes.

28. Scaling Data-Constrained Language Models

    The study distinguishes unique available tokens from total tokens processed through repeated passes. It separately tests fixed unique-data budgets and fixed compute budgets. Repetition can remain useful, but its marginal value declines; some extensively repeated runs develop increasing validation loss. The proposed saturation formula does not model that deterioration. The experiments also compare repetition with adding code and relaxing filtering, treating these as alternative uses of a constrained data budget rather than interchangeable fresh information.

29. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

    This paper changes the optimization question from minimizing pretraining loss under a training budget to minimizing training-plus-inference compute at a specified loss and expected inference demand. Under its model, substantial serving demand favors smaller models trained on more tokens. It also finds that fitting scaling coefficients only at ordinary token-to-parameter ratios overestimates the benefit of additional tokens at extreme ratios.

30. Building AI For All

    Replit describes spending more training compute on a smaller code model to serve latency-sensitive completion, using Chinchilla as motivation for increasing high-quality training data.

31. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    The talk distinguishes quadratic attention computation from linearly growing sequence-dependent memory; solving one does not eliminate the other.

32. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    Fully sharded data parallelism can reduce model memory while leaving attention activations as the limiting allocation.

33. Insights from Snorkel AI running Azure AI Infrastructure

    Choose enough nodes to reach the desired effective batch size, but avoid adding nodes that force an excessively large batch or leave accelerators underutilized.

34. Insights from Snorkel AI running Azure AI Infrastructure

    Treat low accelerator utilization as evidence of a pipeline bottleneck: investigate inter-node networking for multi-node jobs and data loading for single-node jobs, including shared-filesystem read throughput.

35. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    A dedicated GPU fabric allows network topology and performance to be optimized specifically for GPU-to-GPU communication.

36. Saving and Loading Models — PyTorch Tutorials

    PyTorch's inference example instantiates the model structure and restores its saved state dictionary. Its resumable-training example additionally restores optimizer state and records progress such as epoch and loss. Optimizer state contains buffers and parameters updated during training, so restoring model weights alone omits part of the training process. The tutorial separately describes warmstarting from previously learned parameters. It also warns that retaining a state_dict reference without copying or serializing it does not freeze a selected checkpoint.

37. Accelerate v1.7.0: Checkpointing

    Accelerate documents saving and restoring model, optimizer, random-number-generator and gradient-scaler state for training continuation. Custom objects exposing state_dict and load_state_dict can be registered; its example registers a learning-rate scheduler. These saved states are expected to come from the same training script. This supplies a concrete process-state inventory beyond learned weights.

38. TorchData Stateful DataLoader Tutorial

    TorchData's tutorial makes data iteration resumable through state_dict and load_state_dict. Its custom sampler saves both the number of yielded samples and generator state; datasets can also preserve worker-specific random transformation state. For iterable datasets, worker-level states must be captured, and the documented restoration requires the same worker count. The published example compares remaining batches before and after restoration.

39. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

    Pythia makes intermediate model states useful research artifacts by linking checkpoints to exact training-data order and documented hyperparameters. Within each data variant, model sizes share the same ordered stream. Its batch contains 1,024 sequences of 2,048 tokens, and regular checkpoints are spaced by 1,000 updates. The authors provide tokenized data and dataloader reconstruction tools, warning that corpus membership alone cannot establish which material a checkpoint encountered. They also release evaluation code and raw scores rather than relying on comparisons copied from other reports.

40. PyTorch 2.9: Reproducibility

    PyTorch does not guarantee identical results across releases, commits or platforms, or between CPU and GPU execution with identical seeds. Controlling random generators and choosing deterministic algorithms are separate measures. Deterministic algorithms can have performance costs. Therefore preserving logical training state should not be presented as a guarantee of bitwise-identical continuation after changing the execution environment.

41. Training an LLM from Scratch, Locally

    Falling training loss can coexist with worsening held-out loss; validation and generated samples should be checked during training.

42. Data Quality is the Compute Multiplier

    In the Thomson Reuters example, legal midtraining improved domain performance without reported general-capability loss by retaining a majority of data representative of the original pretraining distribution.

43. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples

    The authors construct benchmark derivatives through paraphrasing, translation and code transformations that evade tested overlap detectors. Training on these derivatives raises benchmark scores in their experiments, demonstrating that literal non-overlap need not imply independent assessment. Their proposed detector retrieves embedding-similar candidates and asks an LLM to judge equivalence. The paper also demonstrates false positives from shared multiple-choice answer patterns, distinguishing surface resemblance from the same assessment problem.

44. Language Models are Few-Shot Learners

    Few-shot inference supplies demonstrations as input conditioning while keeping model weights fixed; fine-tuning changes pretrained weights through training. Examples consume bounded context and influence subsequent predictions without becoming parameter updates. Separately, benchmark contamination means evaluation material overlaps training data, weakening claims of generalization to unseen examples. GPT-3's study compares original scores with subsets lacking detected n-gram overlap, but acknowledges false positives and possible distribution differences between clean and original subsets. Conceptually, contamination concerns exposure to evaluation data; optimizing a proxy concerns objective mismatch, while biased reviewer labels concern measurement. Those problems can occur independently.

45. Generalization in Adaptive Data Analysis and Holdout Reuse

    Selecting later analyses using earlier results makes the selection depend on the reused dataset. Aggregate scores can therefore influence selection even when individual test cases remain hidden; ordinary fixed-analysis guarantees no longer follow automatically. Engineering application: after using evaluation failures or scores to revise a coding agent, retain useful failures for regression testing but evaluate the revised configuration on fresh, independently sampled untouched cases for an ordinary holdout claim. Reuse is possible under specialized safeguards: Thresholdout compares training and holdout averages using noisy thresholds, releases controlled answers and stops when its overfitting budget is exhausted.

46. Data Quality is the Compute Multiplier

    Decontaminate training data against downstream benchmarks before interpreting benchmark gains.

47. SWE-rebench: Lessons from Evaluating Coding Agents on Real Software Engineering Tasks — Ibragim Badertdinov, Nebius

    An agent can recover the solution through retained future Git history or public repository access, and disabling one retrieval tool does not close equivalent paths.

48. ChipNeMo: Domain-Adapted LLMs for Chip Design

    NVIDIA continues Llama 2 base models on chip-design code and documentation using autoregressive prediction, then separately adapts them for responses. Tokenizer augmentation requires corresponding embedding initialization. The reported 7B DAPT stage costs 2,620 A100 GPU-hours, excluding original pretraining and later alignment. Domain assessments improve, but the detailed results report slight general-benchmark degradation. A higher learning-rate ablation improves training and validation loss while worsening most task benchmarks. Continuing directly from Llama 2 Chat also degrades response alignment in their experiment.

49. Building AI For All

    Replit's repl-tuned variant continued pretraining on filtered public Replit code, emphasizing languages popular with its users and recent code.

50. Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU

    Use both text-to-item association tasks and masked watch-sequence tasks to adapt a pretrained language model for recommendations.

51. Domain and task adaptation change the training distribution

    Domain-adaptive pretraining continues language-model training on unlabeled text from a relevant domain. Task-adaptive pretraining instead uses the target task's unlabeled training inputs, often a smaller, more specific distribution; task labels are not its prediction targets. Both update parameters and can precede supervised task training. The paper studies masked-token prediction with RoBERTa, not autoregressive instruction tuning. Applied to an autoregressive model, the distinction is the data and supervision contract: continued pretraining predicts ordinary corpus continuations, whereas instruction fine-tuning learns demonstrated responses conditioned on instructions. Both can use token cross-entropy, so the loss function's name alone cannot identify the training stage. This autoregressive comparison is a synthesis with the supplied SFT foundation, not a reported experiment in this paper.

52. Stuffing Context is not Memory, Updating Weights is

    If adaptation preserves the answers supplied by a long context, it could reduce repeated inference context costs and enable ordinary questions without reattaching the corpus.

53. Decoding Mistral AI's Large Language Models

    Instruction tuning uses prompt-response pairs and trains next-token prediction on the response while masking the prompt.

54. The Base Model is Dead

    The speaker proposes distinguishing supervised next-token learning from RL rather than relying exclusively on pretraining, mid-training, and post-training labels.

55. Stuffing Context is not Memory, Updating Weights is

    Restricting updates to a small added parameter set is proposed as a way to reduce catastrophic forgetting while preserving the base model.

56. Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI

    Treat regression constraints as part of optimization rather than checking old behavior only after selecting a fix.

57. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

    The C4 investigation shows how filtering changes available learning material. A word blocklist intended to remove offensive content also excluded documents about science, medicine, law and politics, plus non-offensive discussions of sexual identities. A dialect-model analysis found disproportionate removal of African American English and Hispanic-aligned English. A cleaner-looking corpus can therefore lose useful subject and language coverage.

58. Z.ai GLM-4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

    The described training sequence builds a general language base, adds code and reasoning data, then packs related repository artifacts into longer midtraining contexts.

59. The Base Model is Dead

    Mid-training exposes a model to downstream distributions and longer contexts, allowing agentic traces into the training mixture.

60. Data Quality is the Compute Multiplier

    Avoiding redundant or unnecessary examples can improve learning per unit compute; repeating high-quality data may be preferable to adding low-quality tokens, within a repetition limit.

61. Building AI For All

    Replit built a Spark data pipeline on The Stack, using permissively licensed code and filtering code forms it did not want the model to recommend.