Retrieval-Augmented Generation

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. LangChain Documentation: Retrieval

   LangChain documents a two-step RAG architecture in which retrieval precedes generation and returned information becomes model context. Existing SQL databases, document databases, customer-management systems or internal documentation can supply that information without rebuilding them as a vector database. Its retriever interface returns documents for a query. The documentation separately describes systems where a model chooses when to retrieve. A fixed sequence can bound model-call counts, but retrieval APIs, networks and databases still contribute variable latency.

2. Optimizing LLMs for Speed and Memory

   Model parameters are numerical weight matrices and vectors loaded from a checkpoint; text inputs are represented separately as sequences of vectors. In ordinary inference, request inputs pass through those weights without a training update. Applied to RAG, instructions, conversation history, the question and retrieved text belong to request context when included in the input. Changing that text changes the computation without rewriting the checkpoint. Request-specific cached attention keys and values are intermediate computation state, not newly learned model parameters.

3. Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective

   IBM's Sarah Packowski, Inge Halilovic, Jenifer Schlotfeldt and Trish Smith describe a deployed documentation interface showing ordinary search results alongside a concise generated answer and source links. Their system retrieves complete documentation topics through a separately maintained search API. Before reusing a curated answer, it checks whether its grounding topics changed or were deleted. The paper also illustrates a test-design trap: questions generated from a credentials definition ask what credentials are, while real users may need to know where to obtain them. Passage-generated questions can therefore miss the actual information need.

4. Measuring Attribution in Natural Language Generation Models

   AIS assesses whether an interpretable statement is justified by identified source material. Interpretation includes conversational context and time: a short answer or pronoun must first be resolved into the claim it expresses. Identified evidence may comprise sentences, paragraphs or graph parts, and support can require reasoning across several parts. The paper's examples include deriving age from dates and adding film runtimes; these require valid connecting assumptions as well as source facts. Its annotation process separates interpreting the output from judging attribution.

5. Agent Evals: Finally, With The Map

   Evaluate retrieval correctness and completeness separately from answer faithfulness, question relevance, and factfulness beyond the reference data.

6. Evaluating AI Search: A Practical Framework for Augmented AI Systems

   Evaluate answer completeness, document relevance, and grounding separately to expose different failure dimensions.

7. Building Production-Ready RAG Applications

   Establish a task-specific benchmark and evaluate both retrieval components and the complete query-to-answer pipeline.

8. Specialized RAG Agents: Lessons learned from deploying complex AI systems in production

   Make residual errors inspectable through evaluation, document-linked audit trails, and post-generation checks of claims and attribution.

9. Reading Wikipedia to Answer Open-Domain Questions

   DrQA combined document retrieval with a neural reader that selected answer spans from retrieved Wikipedia paragraphs. Its 2017 contribution addressed a limitation of reading-comprehension benchmarks that supplied the relevant passage in advance: a complete system also had to locate that passage within millions of articles. The retriever used term-based matching, while the reader performed learned text comprehension. The authors deliberately treated Wikipedia as documents without relying on its graph structure. This exemplifies retrieve-and-read question answering before the named RAG formulation, with extraction rather than free-form answer synthesis.

10. NIST TREC: Question Answering Collections

    TREC introduced its question-answering track in 1999, asking systems to return answer-bearing snippets rather than document lists. Responses paired answer text with a supporting document identifier. Subsequent evaluation distinguished a correct answer from an unsupported answer: the latter contained the right response but cited a document that did not establish it. Starting with TREC-9, strict scoring counted unsupported responses as wrong. TREC 2001 also allowed NIL responses expressing that no answer existed in the collection. NIST explicitly warns that its supplied retrieval rankings did not always contain an answer-bearing document.

11. REALM: Retrieval-Augmented Language Model Pre-Training

    Google Research's Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang developed REALM to make external knowledge retrieval learnable during pretraining. Predicting masked text supplies feedback about which retrieved documents help, and the resulting retriever transfers to question answering. Its downstream reader extracts answer spans rather than generating unrestricted explanations. Experiments on Natural Questions, WebQuestions and CuratedTREC reported improved answer accuracy against the evaluated earlier systems. This supplies a complementary historical line: learning access to external knowledge, not merely attaching a fixed search component.

12. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    The original RAG formulation combines a pretrained sequence-to-sequence generator with an external dense index of Wikipedia passages accessed by a neural retriever. It compares conditioning the whole generated sequence on the same retrieved passage with a formulation that can use different passages per token. This establishes the separation between knowledge encoded in model parameters and information retrieved from an external corpus. Modern retrieve-then-prompt applications use the broader idea but do not necessarily implement the paper's jointly trained probabilistic architecture.

13. REPLUG: Retrieval-Augmented Black-Box Language Models

    REPLUG addressed retrieval augmentation when developers could query a language model but could not change its parameters or internal architecture. It prepends retrieved documents to the frozen model's input. Its particular implementation processes documents separately and combines their output probabilities; an optional training procedure adapts the retriever using language-model feedback. This provides a concrete alternative to architectures that train or modify the generator to incorporate retrieval, motivated by API-only access and the expense of adapting large models.

14. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University tested changing answers by replacing the external index without retraining the generator. Using questions about 82 leadership positions whose occupants changed between December 2016 and December 2018, the same Natural Questions RAG model answered 70% correctly for 2016 with the 2016 index and 68% for 2018 with the 2018 index. Mismatched indexes performed substantially worse. The introduction explicitly identifies REALM and ORQA as preceding retrieval-based architectures.

15. Building Production-Ready RAG Applications

    Separate irrelevant retrieved context from missing required evidence; increasing top-K addresses neither problem universally.

16. Retrieval Augmented Generation in the Wild

    Nearest-neighbor retrieval returns candidates even when the corpus cannot answer the query; rank alone does not establish relevance.

17. PROV-DM: The PROV Data Model

    PROV represents artifacts as entities, transformations as activities, and responsible people, organizations or software as agents. Usage links an activity to an input entity; generation links an output entity to its producing activity; derivation connects output to source. Attribution associates an entity with an agent, while association assigns responsibility for an activity. Revision is a specialized derivation. Application mapping: identify each source revision, extracted representation and chunk separately; record the extraction or chunking activity, its inputs, outputs and responsible software. Preserve these relationships when generating embeddings or summaries so a derived artifact can be traced back through intermediate representations.

18. Elasticsearch 8.19 Search API

    Elasticsearch search responses distinguish returned hits from execution completeness. A true timed_out field means results may be partial or empty; shard fields report attempted, successful, skipped and failed work. Hits carry index identity, document identity, relevance score and optionally source fields. Document IDs are unique only within their index. Total-hit metadata distinguishes an exact count from a lower bound. The API can return document versions when requested. Consequently, an empty hit array alone cannot establish that a complete search found no evidence.

19. Document-level access control — Azure AI Search

    Security filters compare application-supplied identity strings with indexed permission metadata; native approaches validate caller tokens against synchronized ACL, RBAC or label metadata. For chunked content, ACL fields belong in index projections. Enforcement uses permissions already stored in the index: source revocations take effect only after the documented synchronization mechanism. Application inference: enforce authorization before model access and across exposed metadata, citations and derived answers. Cached responses need current authorization checks against their source dependencies or invalidation when permissions change; cache partitioning by identity alone does not handle revocation.

20. Anthropic: Introducing Contextual Retrieval

    Exact technical identifiers motivate lexical retrieval alongside embeddings; hybrid/contextual retrieval results do not establish that lexical retrieval always wins.

21. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems

    Context sufficiency asks whether the supplied information permits an answer, without presupposing a reference answer. Topically related information can be insufficient, and combining passages requires the connecting facts rather than an invented relationship. The paper explicitly allows sufficient context to contain an incorrect answer: sufficiency is not source truth. Its experiments distinguish correct answers, incorrect answers and abstentions under sufficient versus insufficient context, including correct answers produced despite insufficient context. Its selective-generation method combines sufficiency and confidence signals rather than treating either as a guarantee.

22. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

    HotpotQA collects questions, answers and sentence-level supporting facts across paired Wikipedia passages. A bridge entity connects a fact in one passage to information needed from another; comparison questions instead require compatible facts about both entities. The paper contrasts asking which band comes from a country with asking whether two bands share a country: the latter requires evidence about both. Its distractor setting supplies two supporting paragraphs alongside eight distractors, whereas its full-wiki setting requires finding evidence without identifying those supporting paragraphs in advance.

23. Building performant RAG applications for production

    LlamaIndex separates representations used to find information from the context used to synthesize an answer. A retrieved sentence can expand into surrounding text; document summaries can link to underlying chunks. Metadata filtering and document hierarchies can narrow retrieval, but their precision depends on useful metadata and structure. These techniques are options to evaluate, not universal prescriptions.

24. FEVER Dataset

    FEVER represents claims with SUPPORTS, REFUTES or NOT ENOUGH INFO labels and records evidence as a list of alternative evidence sets. Each set contains source-page and sentence identifiers. Its published Oliver Reed fixture includes both single-sentence evidence sets and alternatives combining sentences from two pages. This provides a small inspectable example of multiple valid routes to support, rather than requiring one uniquely correct passage. NOT ENOUGH INFO uses null source locations rather than a refuting passage. The distribution supplies a June 2017 Wikipedia snapshot.

25. OWL 2 Web Ontology Language Primer, Second Edition

    The open-world assumption permits an unstated fact to be true: missing information is not automatically false. The OWL primer contrasts this with closed-world interpretation, where absent facts are treated as false. Applied cautiously to RAG, this vocabulary explains why failing to retrieve support does not by itself establish a negative answer. A negative or exhaustive answer needs an additional justification that the relevant information space is complete for the question.

26. Time in XTDB

    System time records when a database version was known to the system; valid time records when the represented fact applies in the modeled world. Late arrivals, retrospective corrections and future-effective changes make these independent. Represent versions with valid-from/to and system-from/to intervals. For half-open intervals, applicability at valid time v and knowledge time s requires valid_from <= v < valid_to and system_from <= s < system_to. XTDB supports valid-time historical queries using current knowledge and system-time queries that reconstruct earlier knowledge. Application implication: ingestion time alone cannot select the applicable policy or fact.

27. Stripe API Reference: Idempotent requests

    Stripe's documented v1-style idempotency mechanism lets a client repeat a create or update request after a connection error without duplicating the operation. The first executed request's status and body are saved, including errors; subsequent requests with the same key return that result. Parameters must match. Keys may be removed after they are at least 24 hours old, and reusing a pruned key creates a new request. Validation failures and concurrent-request conflicts do not save an idempotent result because endpoint execution has not begun.

28. Stripe API v2 overview

    Stripe documents different idempotency contracts for its v1 and v2 namespaces. For v2, replay requires the same idempotency key, the same API, the same account or sandbox, and requests within 30 days. A successful first request causes replay to skip new changes and return an updated response; failed or partially failed requests are re-executed, while an impossible replay returns an error. The page contrasts this with v1's 24-hour replay window and previously saved response. Requests to v2 also require a Stripe-Version header identifying the underlying API version.

29. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    Source authority should influence evidence selection alongside relevance, and the selected source's trust level should remain visible.

30. Mergeable by default: Building the context engine to save time and tokens

    Neither recency nor main-branch code is a sufficient universal truth rule; unresolved conflicts should be surfaced.

31. Integrating Conflicting Data: The Role of Source Dependence

    Copied information can make a false value appear to have majority support across sources. Dong, Berti-Equille and Srivastava therefore model dependence between sources rather than counting every matching report as an independent vote. Their analysis also distinguishes copying from simple agreement: independent sources can share correct values, and a source that copies some records may independently supply others. For answer synthesis, repeated reporting of one upstream assertion should not automatically be described as independent corroboration.

32. DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs

    The paper distinguishes compatible partial information, conflicting opinions or research findings, outdated information, and misinformation. It proposes different responses: reconcile compatible perspectives, expose substantive disagreement, use applicable recent information for changing facts, and reject identified misinformation. Minor differences in wording or precision are not necessarily contradictions. Its evaluation separately measures source support, inclusion of the expected answer and adherence to the conflict-specific response policy. An answer can pass one dimension while failing another, including faithfully repeating an incorrect retrieved claim.

33. GraphRAG: The Marriage of Knowledge Graphs and RAG

    Use vector search to locate initial graph nodes, then traverse relationships to assemble additional context for the LLM.

34. "Data readiness" is a Myth: Reliable AI with an Agentic Semantic Layer — Anushrut Gupta, PromptQL

    Structural relationships do not by themselves define business predicates such as whether a deal is at risk.

35. LangChain 0.0.349: ParentDocumentRetriever

    ParentDocumentRetriever searches smaller child chunks but returns the larger parent material identified by their stored parent IDs. A parent can be an entire original document or a larger section. This separates the unit used to find a match from the unit supplied for interpretation, addressing the loss of surrounding context when only a small matching chunk is returned. The documented implementation stores child embeddings separately from parent documents.

36. Architecting and Testing Controllable Agents

    Decouple the indexed retrieval unit from the context supplied for answer generation.

37. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation

    RECOMP's extractive compressor ranks and concatenates source sentences; its abstractive compressor generates a query-conditioned summary across retrieved documents and can return an empty summary. Training targets downstream usefulness, not merely summary resemblance. Evaluation compares token counts with language-model perplexity and QA exact match/F1. Manual analysis separately judges faithfulness—entailment by retrieved documents—and comprehensiveness—enough information to answer, regardless of origin. A useful summary satisfies both. These axes distinguish unsupported synthesis from missing evidence; downstream accuracy alone cannot. The reported manual analysis samples 30 nonempty summaries per evaluation setting and reveals both failure types.

38. Lost in the Middle: How Language Models Use Long Contexts

    The paper varies evidence position and context length while holding the question and desired answer fixed. Its controlled QA condition includes one answer-bearing document plus distractors; comparisons include no documents and an oracle condition containing only the answer-bearing document. Tested models often perform better when relevant evidence appears near the beginning or end than in the middle. In a separate retrieval experiment, answer performance saturates before retrieval recall. These experiments distinguish finding evidence from successfully using the evidence presented to generation.

39. Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

    Producing a coherent analysis that combines disparate documents is harder than generating variations from one already-understood source.

40. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    Explicitly label estimates and keep their status visible through rewriting and copying.

41. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

    Fusion-in-Decoder investigated how generated answers could combine evidence from multiple retrieved passages. It encodes each question–title–passage combination separately, then lets the decoder attend across the combined representations. This differs from extracting an answer span and from the original RAG model's treatment of retrieved passages. The experiments used both lexical and dense retrieval and found improved answer exact match as passage counts increased within the tested range. The motivation was to supply external knowledge while making multi-passage evidence integration practical.

42. Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

    Decompose broad research instructions into specific subthemes so each output can devote more attention to a narrower subject.

43. Writing Principles for Task-Tuned Prompt Engineering

    Explicitly permit the model to acknowledge insufficient information as a hallucination-reduction tactic.

44. AI Engineering 101

    A context-only prompt can cause abstention when retrieved material is insufficient, even if the underlying model can answer without that restriction.

45. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    A reviewer should be able to navigate from an individual claim directly to its exact supporting paragraph.

46. Claude Platform Docs: Citations

    Claude's citation interface associates response text blocks with source-document locations. Document indices address the request's supplied documents; plain-text citations use character ranges, PDF citations use page ranges, and custom-content citations use block ranges. End indices are exclusive. Plain text and extracted PDF text are sentence-chunked, whereas custom blocks retain their supplied granularity. Only document source content is citable; title and context metadata are not. Anthropic states that the API extracts cited text directly and guarantees valid pointers to supplied documents.

47. Web Annotation Data Model

    A SpecificResource combines a source with a selector. TextQuoteSelector records exact text and optional prefix/suffix for disambiguation; TextPositionSelector uses zero-based, start-inclusive/end-exclusive character offsets in normalized text. DataPositionSelector instead addresses bytes. CSS, XPath and fragment selectors target representation-specific structures. Offsets are brittle under edits. State identifies the relevant representation: TimeState can record sourceDate and a persistent cached copy; HttpRequestState records request headers affecting representation. Application implication: retain source version, representation and normalization identity plus mappings between original and extracted text; do not reuse PDF, HTML and chunk offsets interchangeably.

48. Enabling Large Language Models to Generate Text with Citations

    ALCE separates answer correctness from citation quality. Citation recall asks whether statements are supported by their cited passages; citation precision identifies irrelevant citations. Its automatic checker uses natural-language inference to test whether cited text entails a statement and compares these judgments with human annotations. A citation marker alone therefore is not evidence that the associated claim is supported, and support from a source is distinct from whether that source is correct.

49. Correctness is not Faithfulness in Retrieval Augmented Generation Attributions

    The authors distinguish citation correctness—whether the source supports a claim—from citation faithfulness—whether that source causally contributed to the answer. Identical answer-and-citation pairs can arise through evidence use or post-hoc matching. In experiments with Command-R+ on Natural Questions, appending short answer fragments to other documents sometimes caused those documents to be cited without adding the complete supporting claim. The study therefore tests attribution behavior through interventions rather than treating a plausible citation as proof of reliance.

50. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

    Natural language inference tests whether a premise implies a proposed statement. HANS constructs cases exposing lexical-overlap, subsequence and constituent heuristics that can succeed on ordinary examples while failing on meaning. Its actor-and-judge example reverses who paid whom while retaining overlapping words, changing whether the proposed statement follows. Models trained on MNLI, including the evaluated BERT model, perform poorly on these controlled cases despite stronger conventional benchmark performance.

51. open-rag-eval: RAG Evaluation without "golden" answers.

    Citation faithfulness evaluates the support relationship between a response and its cited passage using graded support categories.

52. Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

    Distill document findings, then verify individual findings in separate calls before synthesizing them.

53. Unanswerability Evaluation for Retrieval Augmented Generation

    Xiangyu Peng and Salesforce Research colleagues' December 2024 UAEval4RAG distinguishes different reasons not to give a direct answer. Its rubric permits clarification for underspecified questions, correction of false premises, and acknowledgment that the knowledge base lacks necessary information. It measures answering, clarification and rejection separately from category-specific response acceptability. Experiments vary retrieval, reranking, rewriting and prompting while evaluating both answerable and synthesized unanswerable questions, exposing tradeoffs between answering successfully and declining appropriately.

54. Architecting and Testing Controllable Agents

    Corrective RAG grades retrieved documents for relevance and uses an alternative retrieval action when they are unsuitable.

55. Selective Question Answering under Domain Shift

    Selective QA answers only when a confidence score exceeds a threshold. Coverage is answered_questions/all_questions; empirical selective risk is wrong_answers/answered_questions, undefined when none are answered. A risk–coverage curve varies the threshold; lower area and greater coverage at a fixed risk are desirable. The paper finds maximum answer probability overconfident on out-of-domain examples. A calibrator trained using in-domain and some known out-of-domain data achieved average coverage of 56.1% at 80% answered-question accuracy, versus 48.2% for maximum probability. Application inference: a verbal confidence statement does not establish calibration; evaluate correctness against confidence and risk–coverage on held-out deployment-like and shifted data.

56. KILT: a Benchmark for Knowledge Intensive Language Tasks

    Fabio Petroni and colleagues' KILT aligns multiple knowledge-intensive tasks to an August 1, 2019 Wikipedia snapshot, with source spans associated with expected outputs. Its 2021 evaluation reports answer performance, evidence retrieval and combined scores separately. Combined KILT scores award downstream credit only when the highest-ranked pages contain a complete annotated provenance set. Alternative evidence sets are supported. Mapping older datasets into one snapshot required checking whether their supporting information remained available and excluding inadequately mapped evaluation examples.

57. Know What You Don't Know: Unanswerable Questions for SQuAD

    SQuAD 2.0 combines answerable questions with adversarially written unanswerable questions about the same paragraphs. Writers made the questions relevant and ensured that the paragraph contained a plausible answer of the requested type. Its published example asks for a treaty's name when the paragraph supplies its year and purpose but not its name, alongside a tempting name belonging to another law. Success requires recognizing when the paragraph does not support an answer and abstaining rather than selecting the most related span.

58. Architecting and Testing Controllable Agents

    Start with a small usable set and expand it using audited, document-grounded synthetic question-answer pairs.

59. RAG Evaluation Is Broken! Here's Why (And How to Fix It)

    Constructing questions around passages with known answers biases evaluation toward local retrieval and can omit realistic aggregation tasks.

60. Building Trust in Enterprise AI: Evaluating Domain-Specific LLMs for Real-World Financial Scenarios

    Context tests should distinguish absent evidence, corrupted document text, and an entirely irrelevant document.

61. RAGChecker: claim-level retrieval and generation diagnosis

    RAGChecker measures whether retrieved chunks entail ground-truth claims, and how many of those retrieved claims appear in the answer. Its faithfulness metric separately checks answer claims against context. These distinguish missing evidence from unused evidence, relative to the reference and entailment judgments. Engineering application: record candidate passages, the exact assembled model context, and the answer separately. Verified support absent from candidates indicates a candidate-retrieval miss; support present there but absent from the assembled context indicates assembly loss; sufficient support in context with a deficient answer indicates a generation-side failure under that input. Check the whole required evidence set, not merely one matching passage. These observations can coexist across claims and do not establish a unique causal explanation.

62. Ragas Faithfulness

    Ragas Faithfulness decomposes the generated response into claims, judges whether each follows from retrieved context, and divides supported response claims by all response claims. Unlike Context Recall, its denominator comes from the response rather than the reference answer. Interpretation: a response can faithfully repeat false source content, so high faithfulness does not independently establish real-world factual correctness. Conversely, an externally correct statement absent from context may receive no faithfulness credit. The metric measures contextual support through a judge, not source authentication.

63. Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

    Ronak Pradeep and colleagues' November 2024 AutoNuggetizer report adapts TREC's earlier nugget methodology to generated answers. It constructs a reference inventory of answer-relevant facts and judges which appear in each response. Its All score divides credited nuggets by the reference inventory size, assigning full, half or zero credit; strict scoring grants only full matches. Missing requested content therefore reduces coverage even when everything actually said is supported. Initial comparisons across 21 topics and 45 runs found strong correlation between fully automated scores and a mostly manual evaluation.

64. Overview of the TREC 2003 Question Answering Track

    Ellen Voorhees's NIST report explains why factoid scoring was insufficient for longer answers. TREC 2003 list questions required distinct answers assembled across documents; assessors expanded reference lists when submissions supplied previously unidentified correct answers. Definition questions instead used information nuggets: assessable facts matched by meaning rather than exact wording, with essential nuggets distinguished from optional details. Repeated nuggets counted once. The report also required appropriate units and correct entity identity when judging answer-bearing passages.

65. Selective Classification: Coverage and Conditional Risk

    A selective predictor combines prediction f with acceptance function g(x) in {0,1}; g=0 means abstention. Coverage is C=E[g(X)], the probability of accepting an input. Selective risk is R=E[loss(f(X),Y)g(X)]/C when C>0: expected loss conditional on acceptance. For N evaluated inputs and A accepted inputs, empirical coverage is A/N; with zero-one loss, empirical selective risk is errors among accepted inputs divided by A. Abstentions remain in the coverage denominator but leave the selective-risk denominator. Consequently lower reported risk can reflect a smaller selected population rather than an improved underlying predictor. Report coverage with risk and examine the risk-coverage curve across selection thresholds.

66. Navigating RAG Optimization with an Evaluation-Driven Compass

    Check source coverage and extraction quality: missing information can cause incomplete or incorrect answers before retrieval configuration matters.

67. Evaluating AI Search: A Practical Framework for Augmented AI Systems

    Access to the retrieved documents used for generation is necessary for the document-based evaluations described; citations alone restricted their use.

68. NIST randomized blocks: applying controlled comparisons to RAG evidence

    NIST describes holding nuisance factors constant within blocks and randomizing remaining variation. Proposed RAG application: for each fixed query, compare the recorded baseline with a candidate-boundary intervention replacing candidates with independently verified sufficient source passages, then run the unchanged reranker, assembler and generator. Separately replace only the final evidence block with sufficient passages, bypassing retrieval and assembly. Keep corpus/ACL/time snapshot, question, prompt template, model/version, decoding settings, token budgets and evaluator fixed; record passage identities, order and every resulting context. For assembly diagnosis, replay the same candidates through original versus evidence-preserving assembly. Match evidence length and position where feasible; otherwise the treatment changes those too. Compare boundary coverage and answer support, not answer wording alone.

69. NIST experimental-design terminology: replication, interactions and confounding

    NIST defines replication as repeating a treatment combination to estimate random error; an interaction occurs when one factor’s effect depends on another. RAG application: repeat each baseline/intervention condition across the same queries with a prespecified sampling policy, randomize execution order, and report paired success-rate changes with uncertainty. Record seeds when available, but a seed is not proof of deterministic service behavior. A reproducible improvement estimates the effect of that defined replacement under the fixed setup, not the sole cause of the original failure. No improvement cannot exonerate retrieval: assembly or generation can still fail. Compare candidate and assembly repairs individually and together when multiple faults are suspected. Final-context substitution bypasses both upstream stages, so its success alone cannot distinguish them.

70. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

    Zhuowan Li and colleagues at Google DeepMind and the University of Michigan compare retrieval-selected context with directly supplied long context across nine English question-based datasets using three 2024 models. Long-context input achieves higher average task scores in the reported comparisons, while retrieval substantially reduces model input. Results differ when source material exceeds a model's context capacity. Their failure analysis identifies questions requiring connecting facts, broad synthesis or implicit relationships that a small retrieved subset can omit.

71. Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

    LinkedIn's system retrieves information from historical support tickets while preserving ticket sections and relationships. Its motivating failure is concrete: splitting a ticket can separate the problem description from its eventual solution. Retrieved ticket information is supplied to a generator, with text-retrieval fallback when graph-query execution fails. In the reported production comparison, the customer-service team was randomly divided between tool use and traditional manual methods. Median issue-resolution time was five hours with the tool versus seven hours without it, a reported 28.6% reduction.

72. Navigating RAG Optimization with an Evaluation-Driven Compass

    Start with a naive baseline and add complexity in response to metric patterns and inspected failures, while keeping the evaluation dataset current.

73. Architecting and Testing Controllable Agents

    Use a data flywheel to turn flagged production failures into curated regression cases.

74. Prompt Engineering is Dead

    Evaluate retrieval, final answers, and context-conditioned generation separately to examine different parts of RAG behavior.

75. CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens

    Similarity alone does not establish entity identity or the explicit relationship chains required by some questions.

76. Building Production-Ready RAG Applications

    More retrieved tokens and reranking do not necessarily improve final response quality; chunk size must be evaluated on the target dataset.

77. Navigating RAG Optimization with an Evaluation-Driven Compass

    In the Qdrant documentation demo, larger chunks increased evidence coverage but reduced faithfulness; retrieving more smaller chunks produced better relevance and faithfulness in the initial comparison.

78. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    Surface disagreements for human examination instead of silently selecting a smoother answer.

79. open-rag-eval: RAG Evaluation without "golden" answers.

    AutoNuggetizer decomposes evaluation into atomic information units and judges their coverage in a generated answer.

80. open-rag-eval: RAG Evaluation without "golden" answers.

    The described HHEM check evaluates the entire response against retrieved content.

81. Architecting and Testing Controllable Agents

    Agentic control flow is useful when the question or intermediate evidence must change the execution path, particularly for routing and self-correction.