Prompting and In-Context Learning

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. Language Models are Unsupervised Multitask Learners

   GPT-2 represents text using byte-level byte-pair encoding: frequent byte sequences become vocabulary tokens, so a token need not equal a word. Autoregressive modeling factorizes sequence probability as pθ(t1,…,tn)=∏i pθ(ti|t1,…,t(i−1)). Generation repeatedly computes a next-token distribution, selects a token, and extends the context. Training estimates parameters θ from text; the paper's zero-shot task execution changes the supplied sequence without modifying parameters or architecture. Instructions, examples, and source text can therefore change the conditional output distribution without constituting a training update.

2. Optimizing LLMs for Speed and Memory

   Model parameters are numerical weight matrices and vectors loaded from a checkpoint; text inputs are represented separately as sequences of vectors. In ordinary inference, request inputs pass through those weights without a training update. Applied to RAG, instructions, conversation history, the question and retrieved text belong to request context when included in the input. Changing that text changes the computation without rewriting the checkpoint. Request-specific cached attention keys and values are intermediate computation state, not newly learned model parameters.

3. Google: Autoregressive models and hallucination

   An autoregressive language model conditions each next-token prediction on preceding tokens, including its own earlier predictions. A generation loop predicts a token, appends it to the context, and repeats. The corresponding probability factorization is P(t1,…,tn|prompt)=∏i P(ti|prompt,t1,…,t(i−1)). Google separately defines hallucination as plausible-looking output that makes a false assertion about the world. Teaching inference: the generation loop has no inherent factual-verification step, so plausible continuation and factual correctness can diverge.

4. Training language models to follow instructions with human feedback

   InstructGPT begins with a pretrained language model. Supervised fine-tuning updates it using demonstrations of desired responses to prompts. Preference training then fits a reward model to human comparisons of candidate responses; PPO updates the response policy to increase predicted reward, with a penalty for departing from the supervised policy. These stages optimize learned parameters using datasets and objectives. Supplying an instruction or tool observation during ordinary inference instead changes the current input to that trained policy. A favorable preference score represents the learned comparison objective, not a proof of factual or program correctness.

5. Decoding Mistral AI's Large Language Models

   Instruction tuning uses prompt-response pairs and trains next-token prediction on the response while masking the prompt.

6. Decoding Mistral AI's Large Language Models

   A next-token pretrained model can possess the needed capability while responding in a format that does not satisfy a human instruction.

7. Prompting best practices — Claude Platform Docs

   Anthropic recommends stating desired outputs and constraints explicitly, explaining the purpose behind instructions, and specifying sequence when order matters. Its demonstration guidance emphasizes examples relevant to the actual use case, varied enough to cover edge cases without teaching accidental patterns, and visibly separated from instructions. The guide treats examples as a way to communicate output format, tone, and structure.

8. Introducing Structured Outputs in the API

   OpenAI describes schema-constrained generation that restricts which tokens may be produced as an output is generated. This adds a mechanism beyond requesting a format in natural language. The report explicitly warns that schema-conforming JSON values can still contain mistakes, such as an incorrect mathematical step, and identifies refusal and incomplete generation as separate outcomes.

9. Cross-Task Generalization via Natural Language Crowdsourcing Instructions

   Mishra, Khashabi, Baral, and Hajishirzi represented 61 tasks and 193,000 input-output instances with structured instructions containing a task definition, command, cautions, unacceptable behavior, and positive and negative examples with explanations. Models were trained on seen tasks and evaluated on disjoint unseen tasks. Instruction components had task-dependent effects: definitions and positive examples often helped, while including negative examples reduced performance in the reported comparison. An instruction is therefore assessable as several distinct obligations, but adding more instructional material is not automatically beneficial.

10. When Will The Benchmaxxing Plague End?

    Contradictory instructions make success impossible, while arbitrary constraints require evidence that performance transfers to real user requests.

11. Language Models are Few-Shot Learners

    The paper distinguishes zero-shot prompting, with a task description but no demonstrations; one-shot prompting, with one demonstration; and few-shot prompting, with several. A demonstration pairs an input with its desired completion, followed by an unanswered input. These categories count examples supplied for inference, not examples encountered during pretraining.

12. Larger language models do in-context learning differently

    This study separates prior label meanings from mappings supplied in the prompt. Replacing sentiment labels with unrelated names such as Foo and Bar requires using the demonstrated association rather than the words' familiar meanings. Some tested models successfully used these mappings, while others relied more heavily on prior semantics. Instruction-tuned PaLM variants improved at unrelated-label mappings but became less willing to follow demonstrations that reversed familiar label meanings.

13. Language Models are Few-Shot Learners

    Few-shot inference supplies demonstrations as input conditioning while keeping model weights fixed; fine-tuning changes pretrained weights through training. Examples consume bounded context and influence subsequent predictions without becoming parameter updates. Separately, benchmark contamination means evaluation material overlaps training data, weakening claims of generalization to unseen examples. GPT-3's study compares original scores with subsets lacking detected n-gram overlap, but acknowledges false positives and possible distribution differences between clean and original subsets. Conceptually, contamination concerns exposure to evaluation data; optimizing a proxy concerns objective mismatch, while biased reviewer labels concern measurement. Those problems can occur independently.

14. AI Engineering 101

    Few-shot prompting can demonstrate output conventions that subsequent requests need not restate.

15. Many-Shot In-Context Learning

    Expanded context windows enabled testing hundreds or thousands of demonstrations. Using the original February 2024 Gemini 1.5 Pro, the study nested smaller demonstration sets inside larger ones and repeated selections across seeds. News-summary performance on XSum improved through roughly 50 examples and then declined, while transfer from XSum demonstrations to XLSum generally improved with more examples. Summaries sometimes fabricated dates or times. More available context therefore enabled useful adaptation without guaranteeing monotonic gains or faithful output.

16. Prompt Engineering & AI Red Teaming

    The speaker favors adding examples but acknowledges that neither a universal example count nor monotonic improvement is established.

17. Building with Anthropic's Claude - The Prompt Doctor is In

    Build curated multi-shot, or few-shot, examples of the desired input-output mapping; pair undesirable outputs with good outputs when teaching subjective distinctions.

18. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    The authors compared no demonstrations, correctly labeled demonstrations, and demonstrations with randomized labels. Across their classification and multiple-choice experiments, randomizing labels often retained much of the benefit of demonstrations. Further ablations identified contributions from the available label names, the input distribution, and the input-output format. In the reported example-count ablation, gains became small beyond eight examples. Demonstration-following performance therefore need not imply that a model inferred the intended input-label rule.

19. Learning to Learn Using Gradient Descent

    Hochreiter, Younger, and Conwell investigated learning to learn: training a system to acquire a procedure for learning related tasks. Their recurrent network received successive function inputs and previous target values, allowing its internal state to incorporate examples. Experiments tested adaptation to changing Boolean, semilinear, and quadratic functions. The motivation was to make learning-algorithm discovery practical with more adjustable parameters than earlier evolutionary approaches.

20. Language Models are Unsupervised Multitask Learners

    GPT-2's summarization experiment appended a summary cue to an article; removing that cue reduced its reported summarization score. Translation supplied example sentence pairs followed by an unfinished pair. These experiments used continuation formats to communicate the intended operation, while the summaries still confused details.

21. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference

    Schick and Schütze's April 2021 PET method reformulates a classification input with a pattern containing one masked position and uses a verbalizer to map task labels to vocabulary words that can fill that position. PET fine-tunes separate masked-language models on a small labeled set, ensembles them to label additional examples, and then trains a final classifier. This was a consequential prompt-based NLP line because authored templates and meaningful label words connected downstream tasks to pretrained cloze behavior, but it combined prompting with parameter training rather than demonstrating inference-only adaptation.

22. Finetuned Language Models Are Zero-Shot Learners

    FLAN addressed weak instruction-only performance by fine-tuning a pretrained language model on tasks expressed through natural-language instructions. Evaluation withheld entire task-type clusters, such as natural-language inference, from instruction tuning. The resulting model improved over its untuned counterpart on unseen task types. Training for instruction responsiveness and supplying an instruction during inference are therefore distinct interventions that can work together.

23. The Power of Scale for Parameter-Efficient Prompt Tuning

    Lester, Al-Rfou, and Constant's November 2021 prompt-tuning method prepends trainable continuous embedding vectors to an input while freezing the base T5 parameters. Backpropagation updates only those prompt vectors. The paper explicitly contrasts this with text-prompt design, which selects ordinary tokens whose embeddings remain fixed, and with full model tuning, which updates model weights. Its experiments found that prompt tuning became more competitive with full model tuning as tested T5 models grew and sometimes transferred better across the evaluated question-answering and paraphrase domains.

24. Decoding Mistral AI's Large Language Models

    The talk identifies prompt engineering, few-shot prompting, chain-of-thought, and retrieval-augmented generation as application adaptation options alongside fine-tuning.

25. Domain adaptation and fine-tuning for domain-specific LLMs

    The speaker recommends treating fine-tuning, prompt engineering, and retrieval as combinable techniques.

26. Coverage-based Example Selection for In-Context Learning

    Selecting examples independently by similarity can fill a prompt with paraphrases while omitting a needed operation. The paper's meeting-scheduling illustration contrasts redundant appointment examples with an example demonstrating how to find someone's manager. Its set-selection method rewards covering different relevant aspects of the query. Eight-shot experiments found the strongest benefits on compositional semantic-parsing splits, which test combining learned operations. The same method did not consistently improve ordinary classification and numerical-reasoning splits.

27. What Makes Good In-Context Examples for GPT-3?

    KATE selects demonstrations by finding training inputs similar to the current query. In a sentiment-transfer experiment, three demonstrations came from SST-2 and predictions were evaluated on IMDB. With temperature zero, random selection averaged 87.95% accuracy with a 2.74-point standard deviation across five runs; selection using an unchanged RoBERTa-large encoder achieved 91.99%. Task-matched encoder training improved results further, whereas other encoder training objectives were less helpful.

28. True Few-Shot Learning with Language Models

    The paper distinguishes the examples placed in a prompt from additional labeled examples used to choose that prompt. When prompt selection was restricted to very few examples, cross-validation and minimum-description-length criteria only slightly outperformed random selection on average and substantially underperformed selection using larger held-out sets. A prompt containing few demonstrations can therefore depend on considerable task-specific development data.

29. The LLM Triangle: Engineering Principles for Robust AI Applications

    Select a small set of relevant demonstrations for each request instead of including the entire example collection.

30. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

    The study held a balanced set of four SST-2 demonstrations fixed and evaluated all 24 orderings across GPT-2 and GPT-3 model sizes. Accuracy varied substantially despite unchanged examples and task. Rankings of effective permutations correlated weakly across models. Increasing demonstration counts improved performance in some settings but did not reliably remove ordering variance. Additional randomly selected demonstration sets were tested to check that the phenomenon was not confined to one set.

31. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design

    The authors varied meaning-preserving formatting choices such as separators, descriptor capitalization, and option numbering while fixing demonstration identity and order. Across 53 classification and multiple-choice tasks, ten sampled formats produced a reported median spread of 7.5 accuracy points across model and shot settings. Formats that worked well for one model could work poorly for another. The study evaluated shared cases and kept demonstrations out of evaluation.

32. Calibrate Before Use: Improving Few-Shot Performance of Language Models

    Experiments identified biases toward labels occurring frequently in demonstrations, labels appearing near the prompt's end, and familiar answer tokens. On balanced SST-2 sentiment data, these biases could make a one-example prompt worse than a zero-example prompt by encouraging repetition of the demonstrated class. The proposed contextual calibration estimates answer preferences with a content-free query and adjusts output scores.

33. Lost in the Middle: How Language Models Use Long Contexts

    Liu and colleagues varied only the position of answer-bearing material within multi-document question-answering prompts and synthetic key-value collections. Several tested models performed best when relevant information appeared near the beginning or end and worse when it appeared in the middle. Adding more retrieved documents could improve recall while answer performance saturated earlier. Repeating the query before and after the context nearly solved the synthetic retrieval task for tested decoder-only models but did little for multi-document question answering. Fitting information inside a supported context window therefore does not establish that its position or added distractors preserve task performance.

34. Transformers: model-specific chat serialization

    A chat template converts ordered role/content messages into the token sequence a causal model continues, inserting model-specific role markers, message boundaries and special tokens. Models fine-tuned from the same base can require different formats: Mistral-Instruct brackets user messages with instruction delimiters, while Zephyr uses explicit speaker markers. The guide warns that incompatible control tokens degrade performance. Where the format requires it, add_generation_prompt=True appends the assistant-start marker; omitting it can cause continuation of the user's message instead of a reply. Some templates need no such marker. Templates already supply required special tokens: use apply_chat_template(tokenize=True), or tokenize the rendered string with add_special_tokens=False to avoid duplicated beginning/end tokens.

35. Claude context windows and cached-token accounting

    Cached prefixes still occupy the context window. System instructions, tool definitions, messages and tool results consume capacity, and generated output also occupies the window. A useful planning constraint is I + O <= W, where I is counted input, O is generated output and W is the model's context capacity. Caching changes processing cost, not this capacity requirement. Inputs exceeding the window are rejected; generation at the limit follows model-specific overflow behavior.

36. Text-generation-launcher arguments

    TGI distinguishes the maximum input length from the maximum total tokens allocated to a request. Clients provide input tokens and request max_new_tokens on top: the budgeting condition is input_tokens + requested_new_tokens <= max_total_tokens, with a separate input limit. For example, a total budget of 1,512 permits 1,000 input tokens and 512 new tokens. Application implication: count the fully serialized prompt, including instructions, history, retrieved evidence and formatting tokens, before reserving output space.

37. Conversation state: managing the context window — OpenAI

    The context window limits tokens used in one request, including supplied input and generated output; applicable reasoning tokens also consume capacity. Instructions, conversation history, retrieved material and tool results supplied to that invocation therefore share its input budget. A simple worked design reserves generated-token capacity before allocating remaining space to input, while also respecting the model's separate output limit. For a reasoning model, reserve space for hidden reasoning as well as the visible answer without counting the same output tokens twice. Persisting a conversation does not make its usable context unbounded.

38. MCP = Mega Context Problem - Matt Carey

    Creating and loading a separate tool for every API endpoint can make the tool descriptions themselves exceed a practical context budget.

39. Effective context engineering for AI agents

    Context engineering selects and maintains the information supplied at each model invocation, including instructions, tools, external data, and conversation history. Instead of loading every possible document, an agent can retain references such as paths or queries and retrieve details when needed; this trades smaller working context for additional exploration latency. Compaction summarizes an existing conversation into a new context, while external notes preserve selected information across calls. Compression can discard details needed later, so shorter context is not automatically better context.

40. Frontier results, on device - RL Nabors, Arize

    Isolate one prompt variable at a time; in this experiment, few-shot examples improved the desired behavior more than input reformatting, strict rules, or explicit chain of thought.

41. Optimization in DSPy

    DSPy's optimization guide starts from a working program and an evaluation procedure, then separates data for optimization, validation, and held-out testing from earlier exploratory development. It warns that prompt optimizers can overfit small datasets. When results remain unsatisfactory, it recommends revisiting task definition, data, metrics, program structure, and optimization choices rather than assuming that another prompt edit will resolve the problem.

42. Why building eval platforms is hard

    Persisting outputs in a nicer UI is insufficient; experimentation requires changing agent configurations and comparing scored runs.

43. Prompt Engineering & AI Red Teaming

    The speaker reports that DSPy outperformed his manual prompt engineering in benchmarks conducted for The Prompt Report.

44. Prompt Engineering & AI Red Teaming

    Reported performance can depend on prompting, output constraints, and answer extraction rather than only on the model.

45. How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

    Maintain tests for edge cases and broader capabilities, use the test set sparingly, and refresh evaluation data with production examples.

46. Shipping AI That Works: An Evaluation Framework for PMs

    Lower judge temperature to improve repeatability and rerun evaluations to characterize remaining variance.

47. Task Fidelity Scaling Laws — Kobie Crawford, Snorkel AI

    Failures must be separated into model capability failures and task or environment defects before low pass rates can be interpreted as meaningful difficulty.

48. Shipping AI That Works: An Evaluation Framework for PMs

    Use 'evals for your evals': compare judge labels with human annotations and inspect disagreements rather than trusting generated explanations alone.

49. NIST randomized blocks: applying controlled comparisons to RAG evidence

    NIST describes holding nuisance factors constant within blocks and randomizing remaining variation. Proposed RAG application: for each fixed query, compare the recorded baseline with a candidate-boundary intervention replacing candidates with independently verified sufficient source passages, then run the unchanged reranker, assembler and generator. Separately replace only the final evidence block with sufficient passages, bypassing retrieval and assembly. Keep corpus/ACL/time snapshot, question, prompt template, model/version, decoding settings, token budgets and evaluator fixed; record passage identities, order and every resulting context. For assembly diagnosis, replay the same candidates through original versus evidence-preserving assembly. Match evidence length and position where feasible; otherwise the treatment changes those too. Compare boundary coverage and answer support, not answer wording alone.

50. How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

    Early 'vibing'—non-scalable, intuition-based inspection—can reveal capabilities and failure patterns before investing in a comprehensive evaluation.

51. How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

    Start with a few core tasks and test prohibited behavior alongside task completion.

52. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Chain-of-thought prompting augments input-answer demonstrations with intermediate natural-language steps. The paper compares this with demonstrations that give answers directly and reports gains on tested arithmetic, commonsense, and symbolic tasks. Its arithmetic experiments reused one set of eight worked examples across several benchmarks, with a separate four-example setup for multiple-choice AQuA.

53. Large Language Models are Zero-Shot Reasoners

    The proposed zero-shot chain-of-thought method requests intermediate steps without worked demonstrations. Its actual experimental procedure has two stages: generate reasoning, then append an answer-extraction instruction and generate the final answer. Experiments used greedy decoding.

54. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

    The authors manipulated answer suggestions and demonstration answer positions while testing GPT-3.5 and Claude 1.0. These changes influenced predictions, yet generated explanations generally omitted the influential cues and sometimes rationalized incorrect answers. The experiment separates an explanation's plausibility from whether it accounts for factors that changed the prediction.

55. Stop Guessing: Build Robust AI with Layered CoT

    The talk treats a generated step-by-step rationale as transparent and verifiable reasoning, but the chunk does not establish that the text faithfully represents the model's internal computation or that visibility alone ensures correctness.

56. Prompt Engineering & AI Red Teaming

    Chain-of-thought prompting, placed under thought inducement in the talk, asks for intermediate steps before the final answer.

57. The LLM Triangle: Engineering Principles for Robust AI Applications

    Few-shot demonstrations can communicate the intended category vocabulary when an instruction admits several correct interpretations.

58. Persona Engineering: A Field Guide to AI Synthetic Personas — Ishan Anand, InsightSciences.ai

    Test whether answer ordering, wording changes, and challenges to stated opinions destabilize the persona.

59. Prompt Engineering & AI Red Teaming

    The speaker argues that deception, misalignment, and reward hacking require intervention below the prompt level.

60. Persona Engineering: A Field Guide to AI Synthetic Personas — Ishan Anand, InsightSciences.ai

    Choose persona construction and model empirically against human ground truth; more detail can amplify bias.