Contents

I — What transfers
II — What supervision teaches
III — Where supervision comes from
IV — Fitting and diagnosing the student
1. Build a coherent training objective
2. Why transfer fails
V — Evidence for replacement
1. Measure capability and fidelity
2. Validate the deployed task
VI — The replacement decision
1. Repay the transfer investment
Check understanding
Open questions
Selected talks
References
Talk library

Distillation: Training Smaller Models From Larger Models’ Behavior

Distillation trains a student model using behavior supplied by a teacher. Instead of paying for the larger model on every request, a team can use its answers, probability distributions or other training signals to develop a student suited to a narrower workload. The student is often smaller, but reduced size is not the definition of the method: the defining step is learning from another model. This chapter explains what to transfer, how to train with it and how to check that the resulting model is useful.

I — What transfers

Learning from another model

Distillation trains a student using supervision supplied by a teacher. Training changes the student’s learned parameters; subsequent predictions can use those parameters without running the teacher. Teacher and student name roles, not mandatory size differences.

The practical opportunity is specialization: learn the part of a teacher’s behavior that a deployment needs. The teacher can be an ensemble, several predictors whose outputs are combined, even when serving that ensemble would be impractical. The investment moves computation into labeling examples and training a reusable student.

Distillation differs from copying weights, which can initialize a model but does not itself teach it from teacher behavior. Caching stores answers for reuse; in-context demonstrations guide a model through its current input. Distillation instead produces a changed checkpoint, a saved model state. Quantization changes numerical precision rather than the source of supervision; a student can be both distilled and quantized.

Capability means performing a specified class of tasks. Behavioral fidelity means matching specified aspects of teacher behavior under stated conditions. These are different goals: reproducing an incorrect teacher answer improves agreement, not correctness. Before choosing a transfer method, specify student-visible inputs, allowed outputs, the intended task population, and acceptable gains and regressions.

For an extraction specialist, a preservation contract could distinguish these requirements.
Property	Required behavior	What may change
Correctness	Recover the right field meanings and values.	Correct a teacher’s mistaken extraction.
Output format	Return the application’s required fields and types.	Omit conversational explanation.
Abstention and refusal	Distinguish insufficient information from prohibited requests.	Change wording while preserving the required decision.
Style	Preserve tone only if the application needs it.	Avoid imitating incidental phrasing.

The supervision interface

The feasible method depends on what the teacher exposes. Returned text supports learning from selected outputs. Scores can support learning from alternatives. Internal access permits matching numerical states inside the model. A written explanation is still an output: receiving it does not expose the hidden computation that produced it.


Teacher artifact	Available supervision	Compatibility requirement
Selected label or text	The particular output returned.	The student must represent the target in its own output format.
Probabilities or logits	Relative preference among alternatives. A logit is an unnormalized output score.	Direct comparison needs corresponding outcomes and prediction positions.
Internal activations	An intermediate numerical representation.	Internal access and a chosen correspondence between teacher and student states.

A soft target is a probability distribution used as supervision. A top-k report contains only k selected alternatives; it is not the complete distribution. The TRL distillation interface, for example, distinguishes full-vocabulary losses from top-k approximations. Check the actual scoring contract rather than assuming an API returns every probability or scores every supplied token.

Different tokenizers can change both vocabulary entries and sequence boundaries, obstructing direct coordinate-by-coordinate probability matching. Teacher-only documents or hints introduce a separate issue: different information at training and deployment. Constructing these inputs or targets produces synthetic examples; making a student learn from them is the transfer step.

Origins of distillation

Teacher–student methods developed around two recurring problems: expensive computation worth consolidating, and useful information that ordinary labels omit. Their history includes internal-state transfer, ensemble compression, and supervision for generated sequences. These approaches address different constraints and can coexist.


Development	Contribution
History Compression — 1992	Jürgen Schmidhuber’s recurrent-network work trained one network to reproduce another’s internal state, potentially making the second network unnecessary. This was an early example of consolidating learned computation, distinct from the later soft-target formulation.
Model Compression — 2006	Buciluă, Caruana, and Niculescu-Mizil trained compact networks on ensemble-labeled inputs to address storage and real-time constraints, including generated inputs when suitable unlabeled data were scarce.
Distilling the Knowledge in a Neural Network — March 2015 preprint	Hinton, Vinyals, and Dean extended compression through teacher probabilities. March dates the posting, not invention.
Sequence-Level Knowledge Distillation — November 2016	Yoon Kim and Alexander Rush adapted transfer to complete translations, using selected teacher sequences to train students.

This development explains why distillation is broader than reducing parameter count. A selected answer, a distribution over answers, and an internal state preserve different information. The next sections examine what each teaches, then turn to where the training examples must come from.

II — What supervision teaches

Selected answers and sequences

A hard target records one selected label or output rather than probabilities over alternatives. With teacher text, supervised fine-tuning, or SFT, increases the likelihood of the recorded response. A token is a model-specific vocabulary unit; a prefix is the sequence preceding a prediction. Under teacher forcing, each prediction receives recorded predecessors, not the student’s earlier sampled choices. The term does not require a live teacher during training. Response supervision develops the underlying objective.

For an abstract recorded completion containing tokens A, B, and C, one sequence supplies three conditional targets.
Prediction	Supplied context	Target
First	Prompt	A
Second	Prompt + recorded A	B
Third	Prompt + recorded A + recorded B	C

The teacher’s selection procedure becomes part of the supervision. Kim and Rush selected translations using beam search, which retains several promising partial continuations. Their sequence-interpolation variant instead selected the candidate most similar to a reference translation. The teacher stayed the same, but the training target changed. Sequence training also concentrated the tested students’ output distributions, helping greedy decoding—choosing the most likely next token—work better in those translation experiments.

Retaining several responses can expose more output variation, while retaining one concentrates exposure. Neither choice recovers the teacher’s complete distribution. Fluent imitation also need not transfer broad competence: The False Promise of Imitating Proprietary LLMs found that persuasive response style could coexist with substantial factual gaps. The unit being copied is observable behavior, including its mistakes.

Probabilities over alternatives

A selected label hides preferences among other alternatives. Matching distributions preserves that guidance. Distillation temperature controls how concentrated teacher and student probabilities are during comparison; raising it can expose useful distinctions or noise.

p_T(i)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\qquad T>0.

Here

z_i

is a teacher logit. Softmax exponentiates scores and normalizes their sum to one; student logits similarly give

q_T

These illustrative teachers share a winner but disagree about the alternatives. Holding logits fixed, temperature-two values normalize square roots of temperature-one probabilities (rounded).
Target	Class A	Class B	Class C
Teacher 1, T = 1	0.6000	0.3000	0.1000
Teacher 2, T = 1	0.6000	0.1000	0.3000
Teacher 1, T = 2	0.4727	0.3343	0.1930
Teacher 2, T = 2	0.4727	0.1930	0.3343

Kullback–Leibler divergence, or KL, measures discrepancy between distributions. Its direction matters because it determines which distribution weights the comparison.

D_{\mathrm{KL}}(p\|q)=\sum_i p_i\log\frac{p_i}{q_i},\qquad D_{\mathrm{KL}}(q\|p)=\sum_i q_i\log\frac{q_i}{p_i}.

Here

p

is the teacher distribution and

q

the student distribution over the same alternatives. Forward KL weights discrepancies by teacher probability; reverse KL weights them by student probability.

Forward KL pressures the student to cover teacher-supported alternatives. Reverse KL penalizes student probability where the teacher assigns little. MiniLLM uses reverse sequence-level KL on student-generated responses. A student unable to reproduce the whole teacher distribution may concentrate on teacher-preferred continuations, but omit others; neither direction universally wins. To isolate the weighting difference, hold a three-class teacher distribution fixed and redistribute the student’s probability. This is a distribution calculation, not a trained-model result.

Which distribution weights the penalty?

Student A: 50%B’s share of remaining mass: 40%

A takes its share first. B receives the selected fraction of what remains; C gets the rest. Every probability stays positive and the sum stays one.

Class A

Teacher p: 0.6000

Student q: 0.5000

Class B

Teacher p: 0.3000

Student q: 0.2000

Class C

Teacher p: 0.1000

Student q: 0.3000

Forward KL, p‖q
0.121171 nats

Reverse KL, q‖p
0.157330 nats

Signed contributions using natural logarithms
Class	Forward: pᵢ ln(pᵢ/qᵢ)	Reverse: qᵢ ln(qᵢ/pᵢ)
A	0.109393 nats	-0.091161 nats
B	0.121640 nats	-0.081093 nats
C	-0.109861 nats	0.329584 nats
Sum	0.121171 nats	0.157330 nats

Individual contributions may be negative; their summed KL is nonnegative. Natural logs give nats. Teacher probability is not correctness, and neither direction universally wins.

Training minimizes a loss, a numerical penalty for mismatching a target. For an independently supplied label $y$ , cross-entropy is $-\log q_1(y)$ : assigning less probability to that label incurs a larger penalty. The subscript one means probabilities are computed at temperature one.

L=(1-\alpha)(-\log q_1(y))+\alpha T^2D_{\mathrm{KL}}(p_T\|q_T).

This combines the label penalty with teacher matching;

0\leq\alpha\leq1

sets their relative weight. For a fixed teacher, KL and teacher-target cross-entropy differ by a constant. The conventional

T^2

approximately compensates for temperature’s effect on gradients—the signals used to update parameters—not teacher error.

Training temperature controls the comparison being learned; sampling temperature controls generated choices. A partial probability report requires an explicit approximation rather than silently treating omitted alternatives as absent. Finally, next-token likelihood is not the probability that a complete answer is true. Calibration requires comparing a specified probability forecast with observed correctness.

Matching internal representations

Outputs supervise the end of a computation. Representation matching adds guidance inside it. A layer is a stage of learned transformation; its hidden state is an intermediate numerical output. Matching selected states requires a correspondence between layers and sometimes a learned projection between dimensions. Equal-width arrays are not automatically comparable, as representation spaces explains.

FitNets, developed by Adriana Romero and colleagues, appeared as a December 2014 preprint and at ICLR 2015. It addressed training deeper, thinner students: a student layer predicts a selected teacher layer’s output through a trainable regressor, followed by output distillation. The hint stage trains the student up to that layer and the regressor. Choosing a guided layer too deep can overconstrain learning.

For coordinate matching, a trainable projection can map two student coordinates to three teacher coordinates. Squared error then compares corresponding values in equal-sized operands. The dimensions and point configurations in the comparison are illustrative; validate any learned guidance against output performance.

Relational Knowledge Distillation instead teaches the student to preserve relationships between examples. Its distance objective divides pairwise distances by their mean within a minibatch—a group of training examples processed together—then compares teacher and student distances. Its angle objective compares the angles formed by triples of examples using normalized difference vectors. These comparisons do not require identical coordinates or even equal representation dimensions. They preserve selected relationships, not every feature or a guarantee of equivalent predictions.

Match coordinates or match relationships

Coordinate targets: project into compatible dimensions

Same input → fixed teacher

Target (1, 2, 0): 3 coordinates

Same input → trainable student

State (1, 1): 2 coordinates

Current trainable projection r(u,v) = (u,v,u−v)

Projected state (1, 1, 0): 3 coordinates

Compare the two 3-coordinate operands: squared error = 1

Loss ⇢ update projection and student through the guided layer. Teacher parameters stay fixed in this hint-training example.

Relational targets: match geometry among examples

Teacher space

A=(0,0); B=(3,0); C=(0,4)

Student space

A=(10,2); B=(10,8); C=(2,2)

A, B and C identify the same three examples in both spaces. The student configuration rotates the teacher triangle 90°, doubles its scale and translates it.

Normalize each distance by that minibatch’s mean pairwise distance
Pair	Teacher distance	Student distance	Normalized, both
AB	3	6	0.75
AC	4	8	1
BC	5	10	1.25
Mean distance	4	8	1

Corresponding angles: A = 90.00°, B = 53.13°, C = 36.87° in both spaces. Matching these relationships does not guarantee equal predictions.

Coordinate matching aligns projected values. Relational matching preserves selected distances and angles while allowing different coordinates; it does not establish equivalent task behavior.

Written reasoning as training data

A reasoning trace contains emitted intermediate steps associated with a proposed solution. Those steps can provide additional learning targets, but they are not hidden states. In hint-intervention experiments, reasoning models sometimes omitted information that influenced their answers. A rationale can therefore be useful training material without being a faithful account of the teacher’s computation.

Rationale supervision creates different deployment contracts.
Training contract	Supervised output	Intended deployment output
Answer only	The answer or label.	Answer.
Trace plus answer	Written steps followed by the answer.	Steps and answer, unless a separately tested policy changes this.
Separate rationale task	Answer prediction and rationale generation are separate tasks.	Answer can be requested without a rationale.

Distilling Step-by-Step, published by Cheng-Yu Hsieh and colleagues at the University of Washington and Google in July 2023, used task prefixes to select label prediction or rationale generation. Its objective, $L_{\text{label}}+\lambda L_{\text{rationale}}$ , weights two tasks; $\lambda$ controls the rationale contribution. It does not require generating a rationale before every deployed answer. Its gains were task-specific, and some settings combined human labels with teacher rationales.

Final-answer checking and step checking establish different things. Consider this deliberately faulty explanation: “ $3\times4=10$ , so the answer is 12.” The answer is correct; the stated intermediate equality is not. Training on the entire completion rewards both. Longer traces also change training exposure and output length, so their value must be separated from simply spending more computation.

DeepSeek’s May 28, 2025 release used the updated R1 teacher’s emitted reasoning chains to post-train Qwen3-8B Base into DeepSeek-R1-0528-Qwen3-8B. This is a separate student lineage, not a lower-precision copy of R1. The Paper Club discussion illustrates this distinction: learning from a teacher trained with reinforcement learning does not mean the student repeats that training procedure. Spending computation on a new problem belongs to Reasoning and Test-Time Compute.

III — Where supervision comes from

Select for the student's workload

A transfer set is the collection of inputs and supervision used to teach the student. Observed requests help represent actual usage; generated inputs can expand coverage. In a function-calling example, combining schema-generated cases with filtered teacher responses to real user inputs worked better than either alone in the reported experiments. That supports testing complementary sources, not a universal mixture.

Select teachers by the students they produce. OpenThoughts reported that QwQ-32B produced stronger students than DeepSeek-R1 under shared sampling settings, despite the latter’s stronger teacher benchmark performance. Its experiments also exposed a tradeoff between more distinct questions and several responses per question. The data-recipe talk recommends testing choices cheaply, then rechecking them at scale.

Selection experiments need to distinguish improved targets from changed exposure.
Comparison	What it revealed	Interpretation boundary
Fine-tune-CoT: Date Understanding, 6.7B student	Checked rationales beat a random subset of equal size; the larger answer-correct set performed best.	The stricter check helped at matched size, but lost useful training volume.
OpenThoughts answer filtering	Random selection beat the tested targeted filters on math; an unfiltered comparison consumed more examples.	Filtering outcomes cannot all be attributed to quality at equal compute.

Inspect difficult cases and teacher–student disagreements rather than assuming they should all be removed. Confidence filtering can omit precisely the region where the student needs instruction; an easy-to-hard curriculum can also fail to address the actual weakness. Keep trusted labels distinct from teacher targets, and use coverage design and filtering analysis to assess what survives.

A useful transfer record identifies the input, teacher version, teacher-visible context, generation and selection settings, retained target, and verification result. Keep rejected candidates traceable when permitted: they explain both selection effects and collection cost. Follow record-level lineage, protect independent assessment, and establish permitted reuse before collecting training material.

Supervise student-visited contexts

A student can fit recorded completions yet struggle after its own different choices. Its policy is the conditional distribution from which choices are made; a trajectory is the resulting sequence of choices and observations. Recorded demonstrations cover their own prefixes. Free-running generation visits the student’s prefixes, including mistakes. Trajectory-based learning explains this feedback between behavior and collected data.

DAgger—Dataset Aggregation—was presented at AISTATS 2011 by Stéphane Ross, Geoffrey Gordon, and Andrew Bagnell. It collects learner-visited states, obtains expert action labels, adds them to an accumulated dataset, and trains again. Collection can mix expert and learner actions. Its driving experiment showed why more expert-only laps need not teach recovery. This was supervised imitation learning, not language-model distillation or environmental reward optimization.

On-policy distillation queries the teacher at prefixes generated by the current student. Generalized Knowledge Distillation mixes these trajectories with recorded sequences and matches teacher–student token distributions. Sampled choices supply contexts, not correct answers. During each update, the sampled sequence is fixed: gradients pass through the distribution-matching loss, not the discrete sampling operation. Refreshing trajectories as the student changes keeps supervision aligned with its behavior.

Compare distributions at the same prefix

Shared initial prompt; same abstract next-token vocabulary

Recorded lane

Prompt + recorded A

Recorded predecessors A supply this lane’s context.

↓ same prefix supplied to both predictors

Teacher: p(· | Prompt, A)

Student: q(· | Prompt, A)

↓ both distributions enter the matching loss

Match at A

Loss ⇢ student parameter update

Student-sampled lane

Prompt + sampled B

Sampling chooses B before this update. Hold B fixed; it is context, not a correct-answer target.

↓ same prefix supplied to both predictors

Teacher: p(· | Prompt, B)

Student: q(· | Prompt, B)

↓ both distributions enter the matching loss

Match at B

Loss ⇢ student parameter update

Solid arrows show data dependencies; dashed update boxes mark gradient-based learning. No gradient crosses discrete sampling and no loss pairs teacher-after-A with student-after-B. This comparison keeps the teacher fixed. After an update, collect fresh student trajectories separately.

Each loss compares teacher and student predictions conditioned on the same prefix. Student sampling selects where supervision is requested, not the correct answer.

Three independent choices are often compressed into the word “online.”
Choice	Alternatives	What changes
Context source	Recorded sequences or current-student trajectories.	Which prefixes receive supervision.
Teacher-query timing	Cached scores or teacher inference during training.	When teacher computation is paid.
Teacher state	Fixed parameters or an explicitly updated teacher.	Whether the supervisor itself changes.

LinkedIn’s training report calls teacher inference during student training “online” even though the teachers remain fixed. Separately, on-policy training may share rollout infrastructure with reinforcement learning while using teacher likelihoods instead of task rewards, as Modern Post-Training explains. More relevant contexts can therefore require more teacher scoring without changing the source of the learning signal.

Teacher-only information

Privileged information is available to the teacher during training but absent from ordinary student inputs. Generalized distillation explicitly permits different teacher and student representations of a case. A richer teacher can supply better targets without requiring the same information at deployment. Whether those targets are learnable from student-visible information remains a separate condition.

This also enables on-policy self-distillation: the same model serves as a better-informed teacher when given a useful hint, while the unhinted student learns from its guidance along student-generated trajectories. The advantage comes from information, not necessarily a larger architecture. Scaling up Continual Learning develops this setup.

The danger is hint leakage: a target relies on a fact the student could not obtain. In the talk’s login example, revealing an expired single-sign-on token can encourage an unsupported diagnosis. Guidance to inspect available logs instead teaches a discoverable procedure. This is a useful distinction, not a guarantee that filtering hints removes shortcuts. If two cases have identical student-visible inputs but require different answers because of a hidden runtime fact, imitation cannot make that fact observable.

For token-level supervision, teacher and student may receive different prompts, but the compared scores must refer to corresponding continuation positions. A hint inserted into one prompt changes offsets; blindly pairing positions can teach the wrong target. Keep the information boundary explicit and test the unhinted student. Context Engineering owns the separate task of assembling information at runtime.

Teacher sees an ordinary login-failure request plus a training-only hint; student sees the ordinary request. Corresponding continuation scores c1 through c3 align despite different prompt lengths. Deployment uses only the ordinary request and an available log-inspection action. — Illustrative login-support case. P and H are schematic prompt and hint segments; horizontal whitespace aligns continuation positions and is not runtime padding. Matching scores updates the student. Guidance can teach an available investigation step but cannot supply a hidden fact before inspection.

IV — Fitting and diagnosing the student

Build a coherent training objective

The recipe must identify a starting checkpoint, trainable components, compatible targets, and exposure budget. Distillation can train a fresh student or adapt an already useful one. Full fine-tuning and Low-Rank Adaptation, or LoRA, are choices about parameter updates, not supervision sources; Post-training explains their difference.

DistilBERT, introduced by Hugging Face’s Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf in October 2019, illustrates initialization followed by learning. It halves BERT’s layers and initializes from alternating teacher layers. Training then combines predicting masked tokens, matching teacher output distributions, and aligning hidden-state directions through a cosine loss. These terms teach complementary aspects of prediction and representation. In the revised paper, random initialization reduced the reported General Language Understanding Evaluation (GLUE) aggregate by 3.69 points. Copying layers supplied a useful starting point, not the complete transfer.

Output, distribution, and representation losses can be combined with explicit weights, but each should serve a tested purpose. TinyBERT separately matches input representations, hidden states, attention-score matrices, and predictions. It tests alternative layer correspondences and removes individual supervision signals in ablation experiments to examine their contribution. Adding a signal adds an assumption about what ought to match; comparing trained students tests whether that assumption helps.

A loss mask selects positions that contribute directly to the penalty. Masking does not remove their text from the conditioning input. The following contract follows Tinker’s multi-turn distillation recipe.
Record region	Available as context	Direct distillation loss
System and user messages	Yes	Excluded
Tool results and assistant headers	Yes	Excluded
Student-generated continuation	Earlier generated tokens condition later ones.	Included

Distinguish task weighting from span weighting. Weighting a separate rationale-generation task does not imply assigning the same weight to rationale tokens inside a trace-plus-answer response. Record exactly which terms and positions are included, which teacher targets are fixed, and whether student parameters or additional projections change. Identical visible examples can therefore produce different updates.

Count candidates generated, targets accepted, and examples or tokens actually consumed separately. Repeated exposure changes the training mixture even when the saved dataset does not change. Select loss weights, exposure, and stopping checkpoints using validation cases and retained-capability checks. Freeze the choice before the final test: repeated adjustment after seeing test results converts that set into development data.

Why transfer fails

Student capacity is the range of behavior its architecture and parameters can represent, not merely its parameter count. A weak result can also arise because suitable behavior was never supervised, inputs lack needed information, targets are misleading, or optimization failed to fit an attainable solution. These explanations call for different interventions.

Use controlled changes to test competing explanations rather than diagnosing from one final score.
Observation	Discriminating intervention
Poor agreement even on training inputs	Check target alignment, optimization, and initialization before attributing everything to generalization.
Good fit on familiar inputs, poor fit elsewhere	Expand missing coverage while holding student and objective fixed.
Different answers require an unavailable fact	Restore the input or narrow the task.
Suspected capacity limit	Compare student sizes under controlled data, supervision, and training conditions.

Cho and Hariharan found that larger, more accurate teachers sometimes produced worse students, supporting a capacity-mismatch interpretation in their classification experiments. Conversely, Stanton and colleagues observed fitting failures even between identical architectures capable of representing the same solution. Teacher strength and theoretical representability do not establish successful learning.

Born-Again Neural Networks demonstrated improved students while retaining the teacher’s architecture in image-classification and language-modeling experiments. Same architecture did not mean copied learned weights, and individual-student results were separate from ensembles of successive generations. Teacher performance is therefore neither a universal ceiling nor a promise that a student will surpass it.

For large reductions, staged transfer is another candidate intervention. The 360Brew team reported better results from progressively smaller students than one large jump. That is a recipe to compare against direct transfer, not a universal requirement. Use failure investigation to decide which experiment addresses the observed weakness.

V — Evidence for replacement

Measure capability and fidelity

Evaluate the teacher, starting student, and distilled student on matched held-out cases. Where trusted labels exist, add a student trained on those labels without teacher supervision to isolate the teacher’s contribution. A common execution protocol helps attribute differences; a second comparison using intended deployment configurations establishes practical usefulness. Do not confuse the two.

For a defined answer-matching rule, agreement and correctness form separate axes.
	Student independently correct	Student independently wrong
Agrees with teacher	Useful agreement.	Shared error; imitation succeeded at the wrong answer.
Disagrees with teacher	Potential correction of teacher error or another valid answer.	Failure requiring inspection of both outputs.

Define fidelity at the required level. Top-label agreement compares winners; predictive KL compares distributions; semantic agreement requires a task-specific interpretation of generated answers. Preserve separate checks for formatting, refusals where relevant, retained capabilities, and changed inputs. The teacher should not be the sole judge: model judges can favor verbosity or familiar output patterns, so validate them independently.

The official DeepSeek-R1-0528-Qwen3-8B model card illustrates why task slices matter.
Evaluation	Distilled Qwen3-8B student	Qwen3-8B comparator
AIME 2024	86.0	76.0
GPQA Diamond	61.1	62.0

The model card permits 64K generated tokens and, where sampling applies, uses temperature 0.6, top-p 0.95, and 16 responses per query to estimate single-attempt success, or pass@1. The comparator is Qwen3-8B, not the Qwen3-8B Base initialization. This is not a clean before/after ablation, and no uncertainty interval establishes the significance of the GPQA difference.

For reasoning transfer, compare answer-only and rationale-trained students with explicit training exposure and inference budgets, recording response length, attempts, and held-out task families. A gain obtained with longer responses may be useful, but it is not evidence of improvement at equal inference work. Reasoning scores also do not establish preservation of safety-critical behavior; those requirements need their own cases and checks.

Validate the deployed task

A replacement claim concerns a complete operating configuration and a defined request population. Distillation need not replace every teacher call: cached results can handle repeated inputs while a student serves uncached requests. The surrounding software still determines what information reaches the model and how outputs are used.

Instacart’s Intent Engine report describes semantic role labeling as extracting product, brand, and attribute concepts from search queries. Offline teacher tags populate a frequent-query cache and train a Llama-3-8B student for uncached, long-tail queries—queries that are individually infrequent. Similar F1 concealed different precision and recall. Meeting latency goals also required adapter merging and upgraded hardware; quantization reduced latency but hurt recall, so the deployed model remained unquantized.

Conceptual reconstruction of the engineering report, not a system screenshot. Cache records are loaded for lookup and the trained checkpoint is deployed for prediction. Hits reuse individual outputs; misses use learned parameters. The common schema does not imply equal accuracy.

Precision measures correctness among predicted tags; recall measures recovery of required tags. F1 is their harmonic mean. Two models can therefore share an F1 while making different omissions and extra predictions. Use an explicit tag-matching policy and keep the components visible rather than treating one aggregate as interchangeability.

The conference account described replacing the extreme-tail fallback with a distilled student as work in progress; the engineering report supplies the production account. Neither supports attributing every system improvement to distillation. Keep the learning change, cache coverage, hardware, and numerical precision separate.

After offline validation, shadow evaluation copies current inputs to a candidate without applying its outputs; bounded live exposure tests consequences that shadowing cannot. Side-effecting tools still require isolation in shadow runs. Follow the live-experiment contract, then benchmark the actual workload. Only validated coverage counts toward the volume available to repay training.

VI — The replacement decision

Repay the transfer investment

Begin with alternatives that meet the same task requirements, including the untrained starting student. If it already suffices, distillation adds avoidable work. AWS’s Bedrock distillation launch account separated teacher generation, student fine-tuning, storage, and provisioned inference charges. Those historical categories illustrate lifecycle accounting, not current universal hosting requirements.

Define a useful lifetime $H$ : the period over which the student and its validated task remain suitable. Let $N$ count eligible requests during that period. Use comparable financial boundaries for the incumbent teacher path and the candidate path, including any incumbent caching. The following is an accounting model, not a published deployment result.


Term	Included work
C_create	All teacher generations, including rejected candidates; verification; training experiments; and evaluation.
C_fixed(H)	Incremental hosting, maintenance, and refresh work over H that is not allocated per request.
c_teacher	Average baseline cost per eligible request under the incumbent completion policy.
c_effective	Student execution, actual output lengths, repeated attempts, and expected remaining teacher work per eligible request.

C_{\text{baseline}}(N)=Nc_{\text{teacher}},\qquad C_{\text{replacement}}(N,H)=C_{\text{create}}+C_{\text{fixed}}(H)+Nc_{\text{effective}}.

With constant average rates and positive per-request savings:

N_{\text{break-even}}=\frac{C_{\text{create}}+C_{\text{fixed}}(H)}{c_{\text{teacher}}-c_{\text{effective}}}.

Equality repays the investment; net savings require greater volume. A nonpositive denominator provides no finite payback for a positive investment.

Assign each charge once. If hosting is already allocated into request cost, do not add it again as fixed cost. If requests outside the validated subset remain unchanged, exclude them from both incremental paths; do not count them as student-saving opportunities. A changing workload can shorten the lifetime even when total company traffic is large.

For an illustrative financial comparison, assume a $200 combined investment, a $0.02 teacher path, and 20,000 eligible requests before refresh. At $0.005 effective student cost, continuous break-even occurs at 13,333.33 requests; 13,334 is the first whole-request count with net savings. If retained teacher work raises effective cost to $0.015, equality occurs at 40,000 requests—beyond the useful horizon; net savings require still greater volume. Both candidates must first satisfy the same quality requirements.

Savings must arrive before refresh

Example

More retained teacher work moves the crossover beyond the same useful-volume horizon.

Effective student cost: $0.005

At the 20,000-request horizon, the baseline costs $400 and the replacement costs $300.

1. Teacher baseline
2. Investment plus effective student cost
3. Useful-volume horizon
4. Break-even
5. Baseline after horizon (hypothetical)
6. Replacement after horizon (hypothetical)

Read coordinates and regions as data

X: 0–45000 requests; Y: 0–1000 USD, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Teacher baseline (polyline)

(0, 0); (20000, 400)

Investment plus effective student cost (polyline)

(0, 200); (20000, 300)

Useful-volume horizon (polyline)

(20000, 0); (20000, 950)

Break-even (points)

(13333, 266.67)

Baseline after horizon (hypothetical) (polyline)

(20000, 400); (45000, 900)

Replacement after horizon (hypothetical) (polyline)

(20000, 300); (45000, 425)

13,333.33: equality: (11000, 340)

Refresh horizon: (21000, 940)

Baseline: (28000, 600)

Replacement: (28000, 300)

Effective student cost: $0.015

Only recurring cost changes. At the horizon, replacement costs $500 against the $400 baseline.

1. Teacher baseline
2. Investment plus effective student cost
3. Useful-volume horizon
4. Break-even
5. Baseline after horizon (hypothetical)
6. Replacement after horizon (hypothetical)

Read coordinates and regions as data

X: 0–45000 requests; Y: 0–1000 USD, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Teacher baseline (polyline)

(0, 0); (20000, 400)

Investment plus effective student cost (polyline)

(0, 200); (20000, 500)

Useful-volume horizon (polyline)

(20000, 0); (20000, 950)

Break-even (points)

(40000, 800)

Baseline after horizon (hypothetical) (polyline)

(20000, 400); (45000, 900)

Replacement after horizon (hypothetical) (polyline)

(20000, 500); (45000, 875)

40,000: equality: (39000, 730)

Refresh horizon: (21000, 940)

Baseline: (28000, 500)

Replacement: (28000, 680)

These invented dollar inputs assume a $200 investment, a $0.02 baseline request, and 20,000 eligible requests before refresh. Only effective student cost changes, from $0.005 to $0.015. Equality occurs at 13,333.33 and 40,000 requests; 13,334 is the first whole-request count with savings in the first case. Dashed cost segments beyond 20,000 show hypothetical continuation. The cited carbon study does not supply these dollar amounts.

Two amortization boundaries deserve separation. LinkedIn’s multi-teacher training report caches outputs by teacher version and data fingerprint for reuse across student experiments. That saves repeated supervision work, not necessarily production request cost. Joseph Attieh and colleagues’ February 2026 translation study separately models production impact plus recurring inference impact. Its carbon accounting supports the same intercept-versus-slope reasoning, but its crossover cannot be converted into dollar payback.

The resulting decision has three legitimate outcomes: replace teacher calls across the validated workload, specialize on an identifiable subset, or retain teacher calls. Cost engineering develops the broader ledger; model routing owns selection and escalation policies. Distillation earns its place when the student preserves what matters and enough useful work remains to justify creating and maintaining it.

Open questions

Predicting teaching utility remains difficult: teacher accuracy does not determine student learning. Progress would mean selecting teachers using transferable diagnostics rather than repeatedly training full students for every candidate.
Rationale supervision needs stronger separation of useful decomposition from extra exposure and longer inference. Controlled comparisons matching training tokens and inference budgets across held-out task families would clarify when traces earn their cost.
Privileged hints can improve supervision while encouraging inaccessible shortcuts. Progress would combine reliable leakage diagnostics with unhinted deployment tests that establish recovery procedures rather than memorized diagnoses.
Targeted adaptation can change unrelated behavior. Reliable preservation of refusal decisions and other consequential capabilities requires coverage beyond reasoning benchmarks, with repeatable regression measurements across successive updates.
Financial replacement claims need complete lifecycle ledgers. Collection failures, verification labor, repeated experiments, residual teacher work, and refresh can dominate savings; prospective accounting through a full useful lifetime would make payback claims more credible.

Cited in this entry

Examine staged distillation and gradual pruning as reported approaches to fitting recommendation-serving constraints, rather than universal compression rules.

Watch talk

Explore more talks

Search the talk library

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

21 matching talks

Talk	Speaker	Event	Year
Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek-R1/V3) — Vibhu Sapra Cited in this entry	Vibhu Sapra	AI Engineer World's Fair 2025	2025
Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code Cited in this entry	Ishan Anand	AI Engineer World's Fair 2024	2024
Low Level Technicals of LLMs Transcript reviewed	Daniel Han	AI Engineer World's Fair 2024	2024
Scaling Compute on Context Transcript reviewed	Jack Morris	AI Engineer World's Fair 2026	2026
How Instacart transformed its search and discovery using an LLM-driven approach Cited in this entry	Vinesh Gudla	AI Engineer World's Fair 2025	2025
How to construct domain-specific LLM evaluation systems. Transcript reviewed	Hamel Husain, Emil Sedgh	AI Engineer World's Fair 2024	2024
Evaling Video Slop Transcript reviewed	Maor Bril	AI Engineer World's Fair 2026	2026
Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI Transcript reviewed	Mahmoud Mabrouk	AI Engineer Europe 2026	2026
Hacking the Inference Pareto Frontier Cited in this entry	Kyle Kranen	AI Engineer World's Fair 2025	2025
Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA Transcript reviewed	Mozhgan Kabiri Chimeh	AI Engineer Europe 2026	2026
20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna AI Transcript reviewed	Bertrand Charpentier	AI Engineer Europe 2026	2026
Bringing Continual Learning into Enterprises Cited in this entry	Samuel Denton	AI Engineer World's Fair 2026	2026
Which Jobs Can Be Replaced Today Transcript reviewed	Fryderyk Wiatrowski, Peter Albert	AI Engineer World's Fair 2024	2024
Scaling Reinforcement Learning: Lessons from Trillion-Token Deployments at Fortune 500s Transcript reviewed	Alessandro Cappelli	AI Engineer Europe 2026	2026
Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel Cited in this entry	Kobie Crawford	AI Engineer Europe 2026	2026
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant Cited in this entry	Jesse Hu	AI Engineer Code 2025	2025
Domain adaptation and fine-tuning for domain-specific LLMs Cited in this entry	Abi Aryan	AI Engineer Summit 2023	2023
How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed Metadata candidate	Ben Kunkle	AI Engineer Europe 2026	2026
Improving Agents is a Data Mining Problem Metadata candidate	Vivek Trivedy	AI Engineer World's Fair 2026	2026
RL Environments at Scale Metadata candidate	Will Brown	AI Engineer Code 2025	2025
You Might Not Need 50 Diffusion Steps Metadata candidate	Ziv Ilan	AI Engineer Europe 2026	2026

References

Coverage and source review

Processed transcripts: 22 processed in full · 5 in the curated path
Automated source review: Passed
Metadata candidates: 4 unreviewed; not verified topic membership
Corpus version: 1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

Build faster, more cost-efficient, highly accurate models with Amazon Bedrock Model Distillation (preview)
AWS announced Bedrock Model Distillation in preview on December 3, 2024; the account records general availability in a May 1, 2025 update. The workflow generates teacher responses and fine-tunes a student, or reuses matching teacher responses from production invocation logs. Its historical billing description separates teacher generation, student fine-tuning, monthly model storage and hourly provisioned inference capacity. It also recommends evaluating the starting student: if that model already performs adequately, distillation adds an unnecessary training step.
Life Cycle-Aware Evaluation of Knowledge Distillation for Machine Translation: Environmental Impact and Translation Quality Trade-offs
Joseph Attieh and colleagues' February 2026 study compares English–Icelandic translation with a 205M-parameter teacher, 65M and 16M students, and student-size models trained without distillation. Its lifecycle accounting separates model production from recurring inference: I(X)=I_prod+X·c_infer. Distillation raises initial impact while a cheaper student can reduce the slope, so savings depend on sufficient served volume. Sequence-target generation adds upfront work without itself changing the deployed student's inference slope. The authors evaluate translation quality alongside impact; the smaller student's cheaper inference does not remove its measured quality deficit.
Distilling the Knowledge in a Neural Network
Distillation trains a student to reproduce information supplied by a trained teacher, which can be a single model or an ensemble. Transfer concerns an input-output mapping rather than copying parameters. Soft targets distribute probability across classes; a hard target selects one class. Relative probabilities among nonwinning classes can convey distinctions discarded by the selected label. Temperature T produces probabilities proportional to exp(logit/T); larger T softens the distribution. The classical objective combines teacher-target cross-entropy at temperature T with independently labeled cross-entropy at temperature 1. Multiplying the soft-target term by T² approximately compensates its gradient scaling when changing temperature. Transfer inputs may be unlabeled. The trained student supplies predictions without running the teacher ensemble.
Model Compression
Buciluă, Caruana and Niculescu-Mizil's 2006 work trains compact neural networks on examples labeled by an ensemble: several predictors whose outputs are combined through averaging or voting. The motivation was that accurate ensembles could exceed storage, computation or real-time constraints. When sufficient unlabeled inputs were unavailable, the authors generated synthetic inputs approximating the relevant data distribution. They explicitly note that teacher labeling and student training can cost more than building the original ensemble; the investment needs justification through deployment constraints or many subsequent predictions.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Hugging Face's Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf introduced DistilBERT in October 2019 as a reusable pretrained student, rather than only a task-specific replacement. It halves BERT's layer count and initializes from alternating teacher layers, then trains with masked-language-model, teacher-distribution and hidden-state cosine losses. Copying weights is therefore an initialization choice, not a substitute for distillation training. Random initialization reduced the reported GLUE aggregate by 3.69 points. The revised paper reports 66 million versus 110 million parameters, GLUE development aggregates of 77.0 versus 79.5, and 410 versus 668 seconds for a complete STS-B development pass on an Intel Xeon E5-2690 v3 CPU at batch size one.
Model Compression via Distillation and Quantization
The paper combines two distinct operations: training a student with a teacher-based loss and representing student weights using a limited set of numerical levels. Its quantized-distillation method incorporates teacher supervision while training quantized student weights. Distillation and quantization can therefore be combined; reducing numerical precision is not itself the act of transferring teacher behavior through student training.
Does Knowledge Distillation Really Work?
The paper separates fidelity—matching teacher predictions—from generalization to independently labeled unseen examples. It measures fidelity using top-label agreement and predictive KL divergence, which compares complete probability distributions. In its self-distillation experiments, greater agreement can accompany lower task accuracy. Students sometimes fail to reproduce teachers even with identical architectures capable of representing the teacher solution. The authors distinguish insufficient transfer-data coverage from optimization failure: the former can leave unseen behavior unspecified, while the latter prevents agreement even on training inputs. Augmentation policies that best improve accuracy need not best improve fidelity; noise and out-of-distribution augmentation hurt the tested models.
NIST AI RMF Playbook: Measure
Construct validity asks whether an indicator measures the concept it claims to measure; external validity concerns generalization beyond development conditions. NIST calls for documented operating conditions, measurement assumptions, limitations and variance. Evaluations using human-subject data should reflect the population in the context of use. Applied to agent evaluation, define the deployment population and scenario dimensions before sampling, document exclusions, and compare sampled conditions with intended users, tasks and operating environments. A split within an unrepresentative dataset does not establish deployment coverage.
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Direct next-token KL matching assumes teacher and student distributions describe the same vocabulary outcomes at corresponding prediction positions. Different tokenizers can change both vocabularies and sequence segmentation, obstructing direct coordinatewise comparison. The paper distinguishes training on teacher-generated text from using output distributions or internal features, and proposes a different objective for cross-tokenizer transfer. Internal-feature supervision additionally requires access to teacher internals.
Reasoning Models Don't Always Say What They Think
The researchers inserted answer hints into evaluation prompts and checked whether reasoning models acknowledged hints that influenced their answers. The tested models often omitted those influences from their written reasoning. Outcome-based training improved acknowledgement initially but did not eliminate the failures. For rationale distillation, the supported implication is that generated explanations are observable supervision, not guaranteed faithful records of the computations or information that produced the teacher's answer.
Attention Is All You Need: token representations and next-token probabilities
Tokens become vectors through learned embeddings, with positional information supplying sequence order. Attention and feed-forward layers transform these representations using learned parameters. Causal masking prevents decoder positions from attending to future tokens. A learned output projection produces vocabulary scores, or logits; softmax converts these into next-token probabilities. In autoregressive generation, the selected token joins the prefix used to predict the following token. Parameters encode learned transformations; the current token sequence supplies request-specific context.
FitNets: Hints for Thin Deep Nets
A FitNets hint is the output of a selected teacher hidden layer. A selected student layer learns to predict that intermediate representation through a trainable regressor whose output matches the teacher layer's dimensions. The hint stage updates the student up to its guided layer and the regressor using squared prediction error; output distillation supplies a separate training stage. This supports students that are deeper and thinner than their teachers. The authors warn that choosing guided layers too deep can overconstrain the student.
TRL v1.7.0: Distillation Trainer
The documented trainer distinguishes exact full-vocabulary loss computation from a top-k approximation. Its external-teacher interface imposes additional restrictions on the supported loss and selected tokens. Consequently, access to a teacher service does not automatically provide the same supervision as local full-distribution access. A top-k report specifies selected alternatives rather than every vocabulary probability; treating it as a complete distribution requires an explicit approximation. The documentation separately controls which completions come from the student and which come from recorded data.
Unifying Distillation and Privileged Information
Generalized distillation allows the teacher to use a privileged representation of each training case while the student receives a different representation available at prediction time. Its procedure trains the teacher, computes soft labels from privileged inputs, and trains the student using ordinary inputs with soft targets and available labels. The paper illustrates the distinction with medical reports versus image pixels. Teacher and student therefore need not receive identical information for transfer to be useful.
Learning Complex, Extended Sequences Using the Principle of History Compression
Schmidhuber's 1992 paper addresses costly sequence learning and difficulty learning dependencies across long time intervals. Its two-network design trains an automatizer on ordinary predictions and on reproducing a second recurrent network's internal state. The second network processes events the automatizer cannot predict. Learning its representations can help the automatizer predict more events, potentially making the second network unnecessary. This provides an early example of transferring learned internal information between networks to consolidate computation.
Distilling the Knowledge in a Neural Network
Geoffrey Hinton, Oriol Vinyals and Jeff Dean's Google paper appeared on arXiv on March 9, 2015; it explicitly extended Caruana's earlier model-compression work.
Sequence-Level Knowledge Distillation
Yoon Kim and Alexander Rush of Harvard published Sequence-Level Knowledge Distillation at EMNLP in November 2016. Their English–German 2×500 student assigned its greedily generated translation 16.9% probability on average after sequence distillation, versus 0.9% for original-data training. This supported their explanation that concentrating the sequence distribution made greedy decoding more effective.
SFT Trainer: objective, shifting, and masking
For prompt x and target response y, supervised fine-tuning minimizes negative conditional log likelihood: L = -sum_t m_t log pi_theta(y_t | x,y_<t), optionally normalized by the count of included tokens. The mask m_t excludes padding and can exclude prompt or non-assistant tokens. Each prediction conditions on the demonstrated prefix, rather than a prefix sampled from the model; this is teacher forcing. The loss rewards probability assigned to the demonstrated next token, not execution success or truth directly. Current TRL supports completion-only and assistant-only loss with dataset/template requirements. Changing masking changes which behavior receives direct supervision, even when the visible example text is identical.
Transformers T5 documentation: Training
Teacher forcing supplies the recorded target sequence as decoder context, shifted so that each position predicts the following target. The documented T5 procedure prepends its start token to decoder inputs and uses target tokens followed by EOS as labels. This establishes the distinction between conditioning training predictions on recorded predecessors and feeding generated choices back during generation. Padding tokens are artificial entries used when batching sequences of different lengths.
Sequence-Level Knowledge Distillation
Kim and Rush approximate a teacher's distribution over complete translations with a selected translation found by beam search. They generate these targets over the training inputs and train the student on the resulting dataset. This selection preserves a chosen continuation rather than the complete sequence distribution. Their sequence-interpolation variant instead selects the beam candidate most similar to the reference translation. Thus decoding and selection rules change the supervision even with the same teacher. Experiments compare these methods with original-data training and token-distribution matching, using separate development and test sets.
OpenThoughts: Data Recipes for Reasoning Models
OpenThoughts compares data strategies by fine-tuning Qwen2.5-7B-Instruct, ordinarily on 31,600 examples per experiment. Sampling several answers per question trades question diversity against response diversity. Its filtering experiment generates 63,200 answers, filters them and samples 31,600 retained pairs; an unfiltered baseline uses all 63,200 and is explicitly not compute-controlled. On math, random selection outperformed the tested targeted filters. Under shared sampling settings, QwQ-32B produced stronger students than DeepSeek-R1 despite weaker teacher benchmark performance. Separate benchmarks were withheld until pipeline selection finished.
The False Promise of Imitating Proprietary LLMs
The study distinguishes task-specific imitation from attempting to reproduce a teacher's broad capabilities. Its imitation models acquired fluent, confident, structured responses that looked competitive in crowd evaluations while factual benchmarks exposed remaining capability gaps. On Natural Questions, the reported LLaMA-13B score was 20 without imitation training, 15 after broad ShareGPT-Mix training and 27 after targeted synthetic-question training, compared with 31 for ChatGPT. More broad imitation data sometimes worsened benchmark performance. Base-model quality and transfer-data coverage therefore affected outcomes separately from stylistic resemblance.
MiniLLM: On-Policy Distillation of Large Language Models
For teacher probabilities p and student probabilities q over the same alternatives, forward KL is Σ p log(p/q), whereas reverse KL is Σ q log(q/p). Reversing the arguments changes which distribution weights discrepancies. Forward KL penalizes failing to cover teacher-supported alternatives; reverse KL penalizes student probability assigned where the teacher assigns little. MiniLLM optimizes reverse sequence-level KL using student-generated responses. Its Gaussian-mixture illustration shows how a restricted student can concentrate on a major teacher mode rather than cover every mode.
Language Models (Mostly) Know What They Know
Calibration compares predicted probabilities with observed correctness frequencies. The paper separates probabilities over explicit multiple-choice labels, self-evaluation of a proposed answer through P(True), and a trained estimate P(IK) that a question can be answered. Formatting and answer alternatives affect calibration: paraphrases distribute probability differently from single-token labels. The studied models can be well calibrated in some task formats, yet calibration deteriorates on other distributions. A high next-token probability therefore cannot simply be read as a probability that an unrestricted completed answer is correct.
TinyBERT: Distilling BERT for Natural Language Understanding
TinyBERT explicitly maps selected student layers to teacher layers for the same input. A learned linear projection maps student hidden states into teacher-width vectors before squared-error matching; another projection handles embeddings. Separate losses match unnormalized attention matrices and task predictions. General distillation supplies an initialization, followed by task-specific distillation using a fine-tuned teacher and augmented examples. Development-set ablations remove individual signals, and layer-mapping experiments compare uniform, upper-layer and lower-layer selections. The best correspondence is therefore an experimentally tested design choice.
FitNets: Hints for Thin Deep Nets
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta and Yoshua Bengio developed FitNets to address the difficulty of optimizing deeper, thinner students. The initial preprint appeared in December 2014; the paper was published at ICLR 2015. Figure 1 distinguishes teacher–student architecture, intermediate-hint initialization and subsequent whole-student distillation. Their image experiments demonstrated students that matched or exceeded teacher accuracy with fewer parameters.
Relational Knowledge Distillation
Relational knowledge distillation transfers relationships between examples instead of requiring each student vector to reproduce its teacher counterpart. Its distance objective matches pairwise Euclidean distances normalized by the mean pairwise distance in each minibatch. Its angle objective matches relationships among triples of examples through normalized difference vectors. These targets can be compared even when teacher and student representation dimensions differ. The selected relationships determine what is preserved, and higher-order relationships can require additional computation.
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
The method extracts teacher-generated labels and natural-language rationales, then trains pretrained T5 students on separate label-prediction and rationale-generation tasks. Task prefixes select which output is requested. Rationales are training targets rather than required inference inputs, removing the need to query the teacher for a rationale at deployment. Experiments vary student size and training-set size and compare against human-label fine-tuning and teacher-label-only distillation. The reported improvements are task-specific across natural-language inference, commonsense questions and arithmetic word problems.
Large Language Models Are Reasoning Teachers
Fine-tune-CoT trains pretrained students to generate teacher rationales followed by answers, initially retaining completions with correct final answers. In Date Understanding, manual rationale checking reduced 170 answer-correct training examples to 123. The reported 6.7B student's test accuracy was 60.36% with all 170 examples, 54.95% with the 123 manually checked examples, and 50.45% with 123 randomly selected answer-correct examples. Stricter checking helped at matched sample count but lost useful training volume. Changing teacher-rationale limits from 128 to 512 tokens produced task-dependent effects and changed student response lengths.
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Cheng-Yu Hsieh and colleagues at the University of Washington and Google published Distilling Step-by-Step in Findings of ACL, July 2023. Its objective is L_label + λL_rationale: separately weighted label-prediction and rationale-generation tasks. Figure 2 shows task prefixes selecting these targets. This weighting is between tasks, not necessarily between spans of one trace-plus-answer completion.
DeepSeek-R1 更新，思考更深，推理更强
DeepSeek's May 28, 2025 release account distinguishes two operations: further post-training the existing DeepSeek-V3-based R1 teacher, and using R1-0528's emitted reasoning chains to post-train Qwen3-8B Base into DeepSeek-R1-0528-Qwen3-8B. The smaller checkpoint is therefore a separate student lineage, not merely the full teacher stored at lower precision.
Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek-R1/V3) — Vibhu Sapra
Reasoning-trace distillation is presented as a way to transfer improvements from a stronger teacher into a smaller Qwen or Llama model without directly training the student with RL.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Rule-based outcome feedback can replace a learned preference reward where correctness is executable: a final math answer can be checked, and code can be run against test cases. The R1-Zero recipe combines accuracy and output-format rewards with GRPO. R1 adds a staged recipe with cold-start demonstrations, reasoning RL, rejection-sampled supervised data, and subsequent RL for broader behavior. Its smaller distilled models receive supervised training on curated outputs rather than inheriting the teacher's RL procedure. This illustrates that the source of a training example and the student's optimization objective are separate choices.
Model-Maxxing: RFT, DPO, SFT (Fine-tuning with OpenAI) — Ilan Bigio, OpenAI
Combine schema-generated examples with filtered teacher outputs on actual unlabeled user inputs.
OpenThoughts: Data Recipes for Reasoning Models
Treat OpenThoughts-style data creation as an experimental pipeline, select promising choices at small scale, and recheck them when scaling.
Statistics Canada Quality Guidelines: Coverage and frames
The target population comprises the units about which information is wanted. Practical exclusions can narrow this to a survey population. A sampling frame identifies and provides access to units; omissions, duplicates, erroneous inclusions and misclassifications create coverage errors. Probability sampling from a restricted survey population does not automatically justify claims about the broader target population. The guidance recommends documenting these differences and periodically checking frame coverage against other sources.
Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel
The reported ablation favored single-table-only training over mixed single/multi-table training and a progressive curriculum; improvements also appeared on the harder multi-table benchmark.
PROV-DM: The PROV Data Model
Provenance records the entities, activities and responsible agents involved in producing or delivering information. Entities include files and document versions; activities use and generate entities; derivation connects an output to an earlier entity through transformation, update or construction. These relationships provide a concrete representation for source-to-derived-data lineage. PROV distinguishes a particular document version from a resource identifying its changing latest version and warns that provenance descriptions must remain valid as resource state changes.
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant
Departures from human-demonstration trajectories can put an agent into unfamiliar states and compound its errors; the talk connects this to DAgger and out-of-distribution behavior.
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
At AISTATS 2011, Stéphane Ross, Geoffrey Gordon and Andrew Bagnell of Carnegie Mellon presented DAgger, or Dataset Aggregation. A learner's actions change its subsequent observations, so expert demonstrations can omit states reached after learner mistakes. DAgger repeatedly collects trajectories under the current policy, obtains expert actions for visited states, adds those labeled examples to the accumulated dataset, and trains another supervised predictor. Collection can mix expert and learner actions while reducing expert participation. In the Super Tux Kart experiment, collecting more expert-only laps did not teach recovery, whereas learner-state aggregation improved driving. The expert supplies action labels; environmental reward optimization is not required by this procedure.
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Sequence-level distillation applies supervised likelihood training to teacher-generated responses. Token-distribution distillation instead minimizes a divergence between teacher and student next-token distributions at each prefix. In fixed-dataset distillation, those prefixes come from teacher or ground-truth sequences. During inference the student visits its own prefixes, including errors, creating distribution mismatch. Generalized Knowledge Distillation mixes fixed examples with student-generated trajectories and queries teacher token distributions at those student prefixes. The implementation differentiates the distribution-matching loss but does not backpropagate through the student's discrete sampling process. It also allows different divergences, useful when student capacity cannot reproduce the teacher distribution.
The Training Infrastructure Behind AI-Powered Job Search: 8X Faster Multi-Teacher Distillation
LinkedIn describes a ranking student supervised by specialized relevance and engagement teachers, combining hard-label cross-entropy with teacher-distribution KL losses. In its online mode, teachers run inference during student training; online does not mean their parameters are updated. Offline mode stores teacher outputs keyed by model version and data fingerprint, reusing unchanged data shards across student experiments. The report gives under two hours for cache generation and under five hours for student training, compared with approximately ten hours for its optimized online path. This amortizes supervision across repeated student experiments.
Scaling up Continual Learning
On-policy self-distillation (OPSD) uses a privileged-information prompt to make the same model a better-informed teacher, then trains the unhinted student against it.
Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect
On-policy distillation uses environment rollouts but scores them using teacher likelihoods rather than the ordinary reward signal.
Scaling up Continual Learning
Hints can teach the student to skip reasoning steps by relying on information unavailable during real execution.
Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect
Give the teacher a hint-enriched prompt, then align its scores back to the original sequence.
Domain adaptation and fine-tuning for domain-specific LLMs
The speaker presents adapter components and selective parameter updates as alternatives to updating every model weight.
Tinker Documentation: Model Distillation
Tinker's multi-turn distillation recipe lets the student call tools, receive results and continue its trajectory. It replaces the environment's task reward with zero and trains using teacher KL supervision. System prompts, user messages, tool results and assistant headers are masked out; only student-generated tokens contribute directly to the loss. The documentation also exposes multi-teacher configurations that associate datasets with teachers and batch sizes, then concatenate their sampled batches. These are explicit choices about supervision and exposure, separate from whether the training infrastructure also supports reinforcement learning.
Cross-validation and held-out evaluation
Testing on data used to fit or repeatedly tune a system can overestimate generalization. The documentation separates training, validation for selection, and an untouched final test set; cross-validation does not eliminate the need for final held-out evaluation. Applied to model judges, tune rubrics and examples on calibration data, freeze the judge, then measure agreement with independent human labels on unseen examples. Applied to context policies, select retrieval and compression settings on development tasks and evaluate the selected policy on held-out tasks. Repeatedly changing settings after seeing test results converts that test set into development data.
On the Efficacy of Knowledge Distillation
Across the studied CIFAR-10 and ImageNet teacher-student combinations, increasing teacher size and accuracy did not reliably improve student accuracy. Some larger teachers produced worse students. The authors compared teacher-student disagreement and distillation loss to investigate whether students matched an unhelpful objective or failed to match the teacher. Their results supported a capacity-mismatch explanation in these experiments. Increasing temperature did not resolve the observed deterioration.
Born-Again Neural Networks
Born-Again Networks train students using teacher output distributions while retaining the teacher's architecture. The paper also studies transfer between different architectures with similar parameter counts. Its experiments demonstrate improved student performance in image classification and language modeling, establishing that distillation need not reduce model size. Successive student generations and ensembles of those generations are separate experimental configurations.
360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI
The team recommends training a strong large model first, then distilling through progressively smaller models rather than making one large size reduction.
Weak supervision needs evaluation beyond supervisor agreement
Oversight becomes difficult when an evaluator cannot reliably judge behavior available to a stronger model: matching its labels can reproduce its mistakes. The paper trains a weak model on ground-truth labels, obtains its predictions on held-out examples, and trains a stronger student on those weak labels. It compares ground-truth test performance with both the weak supervisor and a strong model trained directly on ground truth. Performance gap recovered is PGR=(student-weak)/(strong_ceiling-weak), requiring a nonzero denominator. PGR=0 means no improvement over the supervisor; PGR=1 matches the experimental ceiling. Independent correctness scores and agreement on supervisor-wrong examples distinguish useful generalization from imitation. Experiments show task-dependent generalization and overfitting to weak labels; an auxiliary confidence loss improves some NLP results.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
The paper compares model judges with human preferences and tests position bias, verbosity bias, self-favoring behavior and reasoning errors. Position swaps can change judgments; redundant expansion can receive an undeserved preference; reference answers can help with reasoning tasks. Its GPT-4 judge reached 85% human agreement in one MT-Bench setup excluding ties, versus 81% human-human agreement. For application calibration, collect independent human labels under an explicit rubric, compare judge agreement by task category and tie policy, swap answer order, and test verbosity and model-origin effects. Review disagreements and fix the judge configuration before using its scores for architecture selection.
DeepSeek-R1-0528-Qwen3-8B model card
The model card reports AIME 2024 scores of 86.0 for the distilled Qwen3-8B student and 76.0 for Qwen3-8B, but GPQA Diamond scores of 61.1 and 62.0 respectively. Thus the reported math gain coexists with a lower score on another evaluation. The stated evaluation protocol allows 64K generated tokens and, where sampling is required, uses temperature 0.6, top-p 0.95 and 16 responses per query to estimate pass@1. Deployment guidance says the student's architecture matches Qwen3-8B but requires configuration files from the distilled checkpoint's repository.
Demystifying evals for AI agents
An agent evaluation separates a task and its success criteria from repeated trials, execution transcripts, graders, and final environment outcomes. A booking claim in a transcript is different from an actual reservation in the database. The system under test includes both model and agent harness. Code-based checks suit precise state or test assertions; model graders cover more open-ended properties but require calibration; human review helps establish the standard. Capability suites explore difficult behavior, while regression suites protect behavior that already works.
How Instacart transformed its search and discovery using an LLM-driven approach
A concentrated query distribution supported batch generation and cached serving, with fallback models for uncovered queries.
Building the Intent Engine: How Instacart Is Revamping Query Understanding with LLMs
Instacart describes semantic role labeling as extracting product, brand and attribute concepts from search queries. An offline teacher pipeline produces tags for frequent queries, populates a cache and supplies data for fine-tuning Llama-3-8B. The student handles uncached long-tail queries in real time. The report gives similar F1 but different precision and recall for the production model and comparison model. Meeting its latency target additionally required adapter merging and a hardware upgrade. FP8 quantization reduced latency further but lowered recall, so the team deployed the unquantized model.
Introduction to Information Retrieval: Precision and recall
Precision is TP/(TP+FP); recall is TP/(TP+FN). Applying these set definitions to extraction requires a reference set of field-value pairs, predicted pairs, and an explicit matching policy. With exact pair matching, a wrong value produces both an unmatched prediction (FP) and a missed reference pair (FN); missing pairs are FN and extra pairs FP. Abstaining removes predictions, not required reference pairs, so it can reduce coverage. Zero denominators require an explicit reporting convention.
Scaling up Continual Learning
OPSD supplies vocabulary-wide feedback at each token position without requiring a group of competing rollouts.
Model-Maxxing: RFT, DPO, SFT (Fine-tuning with OpenAI) — Ilan Bigio, OpenAI
SFT teaches an input-to-output mapping through demonstrations, making it useful for classification, formatting, extraction, and small-model distillation.
Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code
The language head applies LayerNorm and an unembedding matrix to produce logits, then the spreadsheet selects the highest-ranked token for its temperature-zero output.
Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect
Score already-generated sequences with the teacher and use the resulting reference log probabilities as the distillation signal.
Hacking the Inference Pareto Frontier
The speaker reports that repeated reconsideration by a smaller model can approach a larger model's quality while retaining a cost advantage.
OpenThoughts: Data Recipes for Reasoning Models
Teacher selection should use student outcomes: the speaker reports Qwen-32B was a stronger teacher than DeepSeek-R1 despite the latter's own benchmark strength.
Bringing Continual Learning into Enterprises
Relevance-masked self-distillation selects task-relevant teacher tokens instead of learning indiscriminately from all token differences.
Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect
An interception server redirects ordinary model API calls into the training inference backend while leaving the harness unaware of RL.
360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI
Alternate small pruning steps with distillation instead of pruning aggressively at the beginning.