Search and Retrieval: Finding the Information a Query Actually Needs

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. A taxonomy of web search

   Broder distinguishes navigational searches, which seek a particular site; informational searches, which seek information; and transactional searches, which seek a place to perform an activity. These needs imply different success criteria: finding one intended destination differs from collecting useful information across pages. The paper explicitly cautions that a query does not reveal its user's intent with certainty.

2. Use the Microsoft Search API to search OneDrive and SharePoint content

   Microsoft Search exposes application search over files, folders, lists, list items, and sites. Requests select entity types and supply a query. Published responses contain ranked hits with summaries, resource identifiers, source URLs, and an indication of whether more results are available. This illustrates a search API as a programmatic interface to selected results, distinct from an interface that enumerates every source record for building another index.

3. In-depth guide to how Google Search works

   Google separates discovering URLs, fetching their content, indexing it, and serving results. Discovery uses sources such as links and sitemaps; there is no central registry of all web pages. Google does not fetch every discovered URL or index every fetched page, and compliance with its guidelines does not guarantee inclusion. JavaScript rendering can be necessary when scripts supply page content. Consequently, a missing search result can reflect an acquisition or indexing boundary before ranking is involved.

4. Building a Smarter AI Agent with Neural RAG

   Search supplies specific information absent from model weights and information that changes after training.

5. RFC 9309: Robots Exclusion Protocol

   A crawler is an automated client; search crawlers recursively traverse links to acquire material for indexing. robots.txt supplies rules that crawlers are requested to honor when accessing URLs. The RFC explicitly states that these rules are not access authorization, and that listing a path can make it publicly discoverable. The September 2022 standard specifies and extends a protocol originally defined in 1994, including matching, caching, and error handling.

6. How To TREC

   A query is an abbreviated expression of an information need; TREC calls the stated need a topic. A test collection combines documents, topics, and relevance judgments. Qrels record which documents assessors judged relevant to each topic. Pooling selects documents for assessment by combining the leading results from submitted runs. This separates the user's need, the query expressing it, system-produced rankings, and the judgments used to evaluate those rankings.

7. The Concept of Relevance in IR

   Borlund’s 2003 research article reviews relevance as multidimensional and potentially dynamic rather than an intrinsic property of a document. It distinguishes system-oriented uses from situational relevance grounded in a user’s current task and context, and argues that interactive retrieval evaluation must account for this changing situation. Consequently, the same document and query can receive different defensible judgments under different tasks or moments.

8. RAG Evaluation Is Broken! Here's Why (And How to Fix It)

   Constructing questions around passages with known answers biases evaluation toward local retrieval and can omit realistic aggregation tasks.

9. Retrieval Augmented Generation in the Wild

   Nearest-neighbor retrieval returns candidates even when the corpus cannot answer the query; rank alone does not establish relevance.

10. Agentic GraphRAG: Simplifying Retrieval Across Structured & Unstructured Data — Zach Blumenfeld

    A document-search agent can mistake its retrieval limit for the total number of matching entities.

11. Information Retrieval from the Ground Up

    Do not convert raw lexical relevance scores into percentages or compare them across queries.

12. Web Annotation Data Model

    The W3C model separates a source resource, a selector locating part of it, and state information identifying the intended representation. A TextPositionSelector uses a start-inclusive, end-exclusive character range after specified text normalization. The standard warns that edits can invalidate these positions and recommends accompanying state information. States are processed before selectors; a TimeState can identify an appropriate time or persistent copy. Retrieval application: an excerpt's offsets need the source representation against which they were measured.

13. Graceful Query Coverage Degradation — Vespa

    Vespa can return results from incomplete query execution when approaching a deadline. Soft timeouts permit reduced coverage; a separate ANN timeout can stop neighbor search early. The response includes coverage and degradation reasons distinguishing timeouts, missing node responses, and configured matching limits. Queueing under overload and query complexity can both prevent completion. A returned list therefore needs execution-status information before it can be interpreted as a completed search.

14. Using Sampled Data and Regression to Merge Search Engine Results

    Federated retrieval searches separately managed collections through their search interfaces. Si and Callan separate describing each collection, selecting which collections to search, and merging returned rankings. Independently operated engines may not expose compatible scores or collection statistics. Their approach uses sampled documents and regression to estimate comparable scores, addressing a problem that cannot be solved merely by concatenating each engine's highest-ranked results.

15. Search — Exa API reference

    Exa's search interface combines a query with result-count controls, domain restrictions, publication-date filters, and optional extracted content. Availability of some filters depends on the selected search category. The current reference marks startCrawlDate and endCrawlDate as deprecated and ignored. Publication-date filtering therefore must not be presented as a working crawl-time freshness control. The interface supplies a bounded ranked response rather than an enumeration of the provider's entire collection.

16. Microsoft Graph throttling guidance

    Microsoft Graph returns HTTP 429 when a request fails because of throttling. Wait the number of seconds in Retry-After before retrying; another 429 requires another delay and retry. When the header is absent, Microsoft recommends exponential backoff. Immediate retries continue consuming usage allowance. JSON batches require inspecting individual responses: the enclosing batch can return HTTP 200 while constituent requests return 429. Retry those failed requests using their individual retry delays; SDK automatic retries do not cover throttled requests inside batches. Connector implication: a scheduled retry or successful batch envelope must not mark failed acquisition as completed synchronization.

17. driveItem: delta — Microsoft Graph v1.0

    An initial delta request enumerates a drive hierarchy through nextLink pages; the final deltaLink retrieves subsequent changes. Deleted items carry a deletion marker. The feed represents latest state rather than a complete event history, and an item can appear repeatedly, requiring its last occurrence to prevail. Track identifiers rather than paths: renaming a folder does not return every descendant. Expired synchronization state can produce HTTP 410 and require fresh enumeration. The documentation recommends initial delta enumeration rather than ordinary folder pagination when constructing a complete local representation during concurrent writes.

18. Building AI Agents that actually automate Knowledge Work

    A document preprocessing layer should synchronize permissions and metadata alongside content, then parse and index it for agent access.

19. Mercator: A scalable, extensible Web crawler

    Mercator's URL frontier holds addresses awaiting retrieval. Workers fetch pages, extract links, resolve addresses, apply scope rules, and enqueue unseen URLs. URL deduplication and content deduplication address different repetition: different addresses can return the same content. Per-host queues help avoid overwhelming one server. The report also describes crawler traps, where generated addresses or session identifiers create effectively unbounded traversal. Fetching, link extraction, and application processing are separate modules; the crawler itself need not be a search index.

20. Sitemaps XML format

    A sitemap lists page URLs with optional metadata that can help a crawler discover and revisit them. Its lastmod field refers to modification of the linked page, not the time the sitemap file was generated. Sitemap indexes can reference multiple sitemap files. Optional metadata communicates publisher-supplied information; search engines can differ in how they use it.

21. RFC 9110: HTTP Semantics

    A client holding an entity tag can send If-None-Match with a conditional GET. If the selected representation still matches, the server can return 304 Not Modified instead of retransmitting its content. This provides a mechanism for checking previously fetched resources without always transferring their bodies. If-None-Match uses weak comparison, so a match need not imply byte-for-byte identity.

22. Downloader Middleware — Scrapy

    Scrapy limits redirect chains with REDIRECT_MAX_TIMES, defaulting to 20; exceeding that limit aborts the request. Meta-refresh redirects share this limit. RETRY_TIMES defaults to two retries beyond the initial download, with a per-request override. Default retryable HTTP statuses are 500, 502, 503, 504, 522, 524, 408, and 429. Configured retryable exceptions include DNS resolution failures, connection failures, timeouts, and response-data loss; exceptions outside RETRY_EXCEPTIONS are not caught by this middleware. Returning another Request schedules future downloading, whereas get_retry_request returns None when retries are exhausted. Scheduling or exhausting attempts therefore does not establish successful acquisition.

23. A Statistical Approach to Mechanized Encoding and Searching of Literary Information

    Luhn's 1957 IBM paper addresses the growing difficulty of organizing and finding technical literature. It proposes statistical treatment of document language as a basis for mechanized encoding and searching, rather than merely accelerating existing manual classification. The proposal connects language analysis, document representation, and matching an information request, providing an early motivation for computing searchable representations from text.

24. Factors Determining the Performance of Indexing Systems, Volume 2

    Cleverdon and Keen's 1966 report compares indexing approaches through retrieval performance rather than assuming that a richer indexing language must work better. Its summary reports advantages for single terms and selected word-form or synonym grouping under the tested conditions, while broader concept grouping could reduce performance. The preface also identifies investigation of retrieval failures as unfinished work, distinguishing aggregate comparisons from explanations of why a system misses useful documents.

25. Relevance Weighting of Search Terms

    Robertson and Spärck Jones’s 1976 paper derives term-weighting methods from a probabilistic retrieval theory. It treats ranked retrieval as selecting relevant documents from many nonrelevant documents and uses both collection distribution and known relevance information to weight discriminating query terms. The work established a probabilistic line of retrieval that later informed Okapi scoring; it did not claim that estimated relevance is certain or independent of the query and collection.

26. Okapi at TREC-3

    The Okapi team describes successive changes to a probabilistic retrieval system: incorporating within-document term frequency, document length, and query frequency, then combining two earlier weighting variants into BM25. The combination moderated competing assumptions about how document length should affect a term's contribution. This places BM25 within iterative experimental development rather than presenting it as an isolated formula that replaced all previous retrieval methods.

27. The Anatomy of a Large-Scale Hypertextual Web Search Engine

    Brin and Page's Google paper supplements textual matching with the web's link structure. PageRank weights incoming links using properties of their source pages rather than simply counting them. Anchor text—the words used in a link—is associated with its destination, allowing other pages to describe a resource, including resources whose own text is unavailable. These signals complement document text instead of eliminating lexical retrieval.

28. RankNet: A ranking retrospective

    Burges describes learning a ranking score from features of query-document pairs and human relevance judgments. RankNet, introduced in 2005, learned pairwise ordering, but treating ordering errors similarly did not adequately emphasize mistakes near the top of a list. LambdaRank addressed alignment with ranking measures, while LambdaMART combined that approach with boosted decision trees. The development illustrates separate choices about training signals, list objectives, and the model used to compute scores.

29. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods

    Cormack, Clarke and Büttcher introduced Reciprocal Rank Fusion at SIGIR 2009 as an unsupervised way to combine rankings. A document receives 1/(k+r) from each input list in which it appears, and contributions are summed. The authors fixed k=60 after pilot work and evaluated fusion across several TREC result sets and LETOR 3. In those experiments RRF generally exceeded the individual rankings and the tested Condorcet and CombMNZ alternatives.

30. Passage Re-ranking with BERT

    Nogueira and Cho adapted a pretrained language model to judge query-passage relevance using labeled retrieval examples. Their experiments retained BM25 for first-stage retrieval and applied BERT to its shortlist, reporting improved MS MARCO ranking results. The paper also identifies questions whose relevant passage never enters that shortlist. The contribution therefore strengthened candidate ordering while leaving candidate-generation misses as a separate limitation.

31. Dense Passage Retrieval for Open-Domain Question Answering

    DPR uses separate encoders for questions and passages, then scores a pair by the dot product of its vectors. Passage vectors can be computed and indexed offline; a new question is encoded at request time and used to retrieve high-scoring passages. Training contrasts relevant passages with negatives so similarity becomes useful for the retrieval task. This factorization explains efficient candidate retrieval and its limitation: a similarity score is a learned ranking signal, not a probability that a passage proves the answer. Lexical retrieval such as BM25 instead exploits matching terms and remains a useful comparator.

32. Chunk large documents for RAG and vector search — Azure AI Search

    Chunking can use fixed sizes, sentence or section boundaries, or combinations. Heading markup can define section boundaries, and adding a document title to an interior chunk can preserve context that would otherwise be absent. Overlap repeats content across neighboring chunks to maintain continuity. The documentation notes that even a document fitting an embedding model's input limit can benefit from finer retrieval units when it covers several subtopics.

33. Chunking — Docling

    Docling chunkers produce document portions accompanied by metadata. HybridChunker refines hierarchical chunks using a supplied tokenizer: it splits oversized chunks and can merge consecutive undersized chunks sharing headings and captions. Its table option repeats headers when a table spans multiple chunks. This supplies a concrete mechanism for keeping independently retrieved table portions connected to their column meaning while respecting a token budget.

34. Information Retrieval from the Ground Up

    Chunking limits the context each vector must represent and makes it possible to locate and link to the relevant passage.

35. Define an index projection for parent-child indexing — Azure AI Search

    Index projections distinguish each chunk's unique key from its parent identifier and can repeat parent fields on chunks or place parent metadata in a separate index. Separate-index retrieval requires application-side lookup and collation because query-time joins are unsupported. Generated chunk keys include a hash that changes when the parent is updated, so these keys are not permanent source identities. Deleting a parent removes its children only when the configured deletion-detection mechanism observes the deletion; otherwise explicit child deletion is required.

36. Collapse search results — Elasticsearch

    Field collapsing selects the highest-sorted hit for each grouping key, while inner_hits can expose additional members of that group. The response's total hit count remains the uncollapsed document count, not the number of distinct groups. Expanding groups performs additional queries and can increase latency. Applied to passage retrieval, grouping by a chosen source key can show one source with several passages while keeping source counts distinct from passage counts.

37. Introduction to Information Retrieval: A first take at building an inverted index

    An inverted index maps normalized terms to documents containing them. Construction assigns each document a docID, emits term–docID pairs, sorts them and merges repeated occurrences within a document. A dictionary stores each term, its document frequency and access to its postings list; postings are ordered by docID. Query-term lookup therefore yields candidate identities without scanning all document text. Boolean conjunction can intersect lists and disjunction can union them. Frequencies and positions can enrich postings for scoring and phrase matching, but identifying a matching document is distinct from ranking it or determining whether it supports an answer.

38. Index and search analysis — Elasticsearch

    An analyzer is a set of text-processing rules applied when indexing fields or interpreting queries. Compatible processing allows differences in capitalization and word forms to match. Identical analyzers are common but not mandatory: Elastic's prefix-search example indexes word prefixes while leaving the query as one token. Expanding the query into prefixes too would incorrectly match shorter shared prefixes. Compatibility concerns intended matching behavior, not simply using the same analyzer name.

39. Information Retrieval from the Ground Up

    An inverted index moves token extraction and lookup preparation to ingestion; stored offsets and positions support highlighting and phrase checks.

40. Introduction to Information Retrieval: Positional indexes

    Phrase retrieval first finds documents containing the query terms, then checks whether their stored occurrence positions satisfy the required arrangement. Merely intersecting document identifiers establishes co-occurrence, not a phrase match. The same position information supports proximity conditions such as two terms appearing within a specified number of words.

41. Query and filter context — Elasticsearch

    Query context both determines matches and computes scores; filter context evaluates a yes-or-no condition without contributing a relevance score. Elastic's example combines scored title and content matches with mandatory publication-status and date-range filters. This provides a concrete implementation of separating eligibility from ordering: a result cannot compensate for failing a required filter by scoring highly elsewhere.

42. Retrieve & Re-Rank — Sentence Transformers

    A retrieve-and-rerank pipeline first selects a candidate set with lexical search or a bi-encoder, then jointly processes each query-candidate pair with a cross-encoder. Joint scoring is expensive across a whole large collection, motivating a bounded shortlist. The documented Wikipedia workflow retrieves paragraphs and returns the five highest-scoring reranked passages. Candidate depth and returned-result count are thus separate choices. Because reranking operates on supplied candidates, it cannot recover a passage omitted from that input.

43. Magic WAND: Faster Retrieval of Top Hits in Elasticsearch

    Lexical top-k retrieval can skip documents whose maximum possible score cannot exceed the current kth-best score. WAND uses upper bounds on query-term contributions; Block-Max WAND tightens those bounds for individual blocks of postings. The implementation report distinguishes accurate top-ranked hits from exhaustive hit counting: skipping noncompetitive matches preserves the former but prevents automatically obtaining the latter. Its published example shows why documents matching only a low-contribution term become unnecessary to visit once the competitive threshold rises.

44. Efficient Query Evaluation Using a Two-Level Retrieval Process

    Broder, Carmel, Herscovici, Soffer and Zien’s 2003 WAND paper separates preliminary candidate identification from full scoring. Query-term posting lists and score bounds allow large portions of the index to be skipped. Safe optimization preserves the exact top results; more aggressive approximate pruning can create false negatives. In the reported TREC Web Track experiments, safe settings reduced full evaluations by more than 90%, while further reductions accepted some effectiveness risk.

45. Introduction to Information Retrieval: Okapi BM25

    BM25 ranks documents using query-term matches with inverse document-frequency weighting, term-frequency scaling and document-length normalization. Repeated occurrences do not contribute indefinitely in direct proportion to count; tuning controls saturation and how much document length matters. This gives lexical retrieval a concrete mechanism different from learned vector similarity. The source recommends tuning on development relevance judgments rather than selecting parameters using the final test collection.

46. Information Retrieval from the Ground Up

    Lexical ranking combines term frequency, corpus rarity, and field-length normalization; equal term counts do not imply equal scores.

47. A statistical interpretation of term specificity and its application in retrieval

    Spärck Jones argues that a term's specificity can be estimated from how many collection documents contain it. Frequent terms can generate irrelevant matches, but simply deleting them can also lose relevant material. Weighting less widespread terms more heavily offers another choice. The paper distinguishes collection-level specificity from within-document frequency, which can indicate how prominently a document discusses a subject.

48. Information Retrieval from the Ground Up

    Use Boolean filters for hard eligibility and rank features for preferences among eligible results.

49. Keyword type family — Elasticsearch

    Identifiers such as ISBNs and product IDs commonly require term-level lookup rather than numeric range comparisons. Elasticsearch supports representing them as keyword fields even when they contain digits. Field configuration can also explain missing candidates: values exceeding ignore_above remain unavailable for keyword matching, including values omitted from dynamically created keyword subfields. A source value's presence therefore does not establish its presence in every searchable representation.

50. Information Retrieval from the Ground Up

    Stop-word removal can erase an entire meaningful phrase or discard negation, so its usefulness depends on the retrieval task.

51. Information Retrieval from the Ground Up

    Use language-appropriate analyzers and separate language fields or indices to avoid applying incorrect rules and mixing frequency statistics.

52. E5-base-v2 model card

    E5-base-v2 specifies different input prefixes for queries and passages in asymmetric retrieval. Its authors state that omitting these training conventions degrades performance. The published implementation uses attention-mask-aware average pooling, L2 normalization, and dot products between normalized embeddings. It truncates inputs to 512 tokens, and the model card limits applicability to English. These requirements make input role, preprocessing, pooling, length limits, and comparison conventions concrete parts of a retrieval interface.

53. pgvector README

    pgvector performs exact nearest-neighbor search by default; approximate indexes trade neighbor recovery for speed and can change query results. IVFFlat divides vectors into lists and searches selected lists closest to the query. Increasing probes searches more lists at additional cost. The documentation contrasts IVFFlat's lower memory and faster construction with HNSW's graph-based speed–recall tradeoff and lack of an index-training step. Evaluation implication: approximate-search recovery should be checked against exact neighbors using the same stored vectors, distance function, and eligible records.

54. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms

    ANN-Benchmarks measures index construction time, index size, query time, and recovery of exact vector neighbors separately. Its distance-based recall accepts returned points within the exact kth-neighbor distance, avoiding arbitrary penalties when distances tie. Runs preserve algorithm parameters, returned identities, distances, and individual query timings. Published comparisons vary datasets and requested neighbor counts; the fastest method changes with the desired recall. Its plots retain nondominated speed–recall configurations, while scatter plots expose configurations hidden by that summary.

55. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

    HNSW organizes vectors in a hierarchy of proximity graphs. Search moves greedily through sparse upper layers and explores a larger candidate set near the bottom, avoiding exhaustive comparison with every vector. Graph connectivity and candidate-list breadth affect resource use and recall. The returned neighbors are approximate under the chosen distance function: index recall and the semantic relevance of the embeddings are separate sources of retrieval error.

56. Product Quantization for Nearest Neighbor Search

    Jégou, Douze and Schmid’s journal paper, published in 2011 after an August 2009 technical report, divides a vector into low-dimensional subvectors and quantizes each subspace separately. The resulting short code supports approximate distance estimation, including asymmetric comparison between an uncompressed query and compressed database codes. Combining these codes with an inverted-file system avoids exhaustive comparison and trades memory and computation against exact-neighbor recovery.

57. Vector Query Filters — Azure AI Search

    Azure's preFilter applies predicates during HNSW traversal on each shard and can require more traversal for restrictive filters. postFilter first selects unfiltered candidates per shard and then filters them before global merging. strictPostFilter filters only the unfiltered global top-k. The latter approaches can omit eligible neighbors or return no results even when matching records exist outside the searched shortlist. Candidate depth and filter placement therefore affect both resource use and result coverage.

58. Vector Search Benchmark[eting]

    For HNSW approximate nearest-neighbor search, filtering can increase search work even though fewer records qualify.

59. Vector Search Benchmark[eting]

    Compare approximate nearest-neighbor performance alongside precision and recall, with search parameters disclosed.

60. Layering every technique in RAG, one query at a time

    Ranking needs domain constraints, corpus structure, and user preferences in addition to relevance—the speaker distinguishes horizontal semantics from vertical semantics.

61. Introduction to Information Retrieval: Near-duplicates and shingling

    A k-shingle is a contiguous sequence of k terms; each document becomes a set of such sequences. Jaccard similarity is |A intersection B|/|A union B|. A threshold identifies substantial textual overlap. MinHash estimates this similarity because, under a random permutation, the probability that two sets share the minimum element equals their Jaccard similarity. Exact fingerprints and approximate overlap serve different purposes. Application inference: keep document identity, source location, revision identifier, content hash, validity and provenance separately from duplicate-cluster membership. A small change to a date, number or negation can be material despite high overlap.

62. Summarization: (1) Using MMR for Diversity-Based Reranking and (2) Evaluating Summaries

    Maximum Marginal Relevance selects results by balancing relevance to the query against similarity to results already selected. Its motivation is that a list can contain individually relevant documents yet repeat the same information. The paper's retrieval example demotes redundant reports while promoting relevant material that adds something different. Because redundancy depends on the selected set, this is a list-level decision rather than independent relevance scoring alone.

63. Combating Web Spam with TrustRank

    Gyöngyi, Garcia-Molina, and Pedersen’s 2004 paper identifies link manipulation as an adversarial ranking problem: operators can create many bogus pages pointing to a target because incoming-link signals can raise its rank. The proposed TrustRank mechanism does not detect every manipulation directly. Human experts first identify reputable seed sites; a biased PageRank computation then propagates dampened trust through outgoing links. The resulting score can filter an index or supplement PageRank and other ranking signals. In the authors’ AltaVista-based experiment, TrustRank separated reputable from spam sites better than the tested PageRank and uninformed baselines.

64. Bypassing the Multimodal Tax: Framework-Free Hybrid RAG, Raw SQL RRF, and Live UI Telemetry

    Semantic proximity is insufficient when queries depend on exact product names, identifiers, or other specific terms; the speaker recommends hybrid retrieval.

65. Benchmarking semantic code retrieval on Claude Code

    The task-level breakdown favored semantic search for behaviorally related files with different keywords, and grep for import tracing when useful keywords were found early.

66. Running AI Application in Minutes: Quick Start with AI Templates

    In the reported Azure AI Search experiments, combining vector and text results alone underperformed text-only retrieval; adding semantic reranking produced the best results.

67. Reciprocal rank fusion — Elasticsearch

    Elasticsearch distinguishes the number of results admitted from each child retriever from the final response size. rank_window_size bounds each input list, while size limits the fused output. A vector retriever returning fewer than the window contributes only its available results; one returning more is truncated before fusion. The published example combines two global top-50 lists and returns ten results. Changing the output size therefore cannot recover contributions already excluded by an upstream window.

68. Normalization processor — OpenSearch

    OpenSearch provides score fusion by normalizing each subquery's scores and combining them with a selected rule. With min-max normalization and no explicit bounds, the extrema come from retrieved results. Arithmetic combination supports per-query weights summing to one; the documentation supplies a 30/70 example. Consequently, normalization settings and retrieved score ranges are part of the fusion contract, alongside weights. The processor also documents that its failure fails the pipeline rather than silently returning an unfused list.

69. Relevance scoring in hybrid search using Reciprocal Rank Fusion — Azure AI Search

    Hybrid retrieval can combine independently ranked lexical and vector results. Reciprocal Rank Fusion assigns a document a contribution of 1/(rank + k) in each list where it appears and sums those contributions. It merges ranks rather than directly comparing BM25 and vector score magnitudes. The fusion constant k is different from a nearest-neighbor result count. Azure's semantic reranker runs after fusion and reports a separate score. Fusion, candidate count, and reranking are therefore distinct controls with different effects on the evidence set.

70. Layering every technique in RAG, one query at a time

    Cross-encoders can resolve competing retrieval candidates through joint query-document scoring, but their expense motivates a retrieve-then-rerank pipeline.

71. The RAG Stack We Landed On After 37 Fails

    Use a bi-encoder for broad candidate retrieval and a cross-encoder, also called a re-ranker, for a smaller post-retrieval candidate set.

72. MS MARCO Cross-Encoders — Sentence Transformers

    The published cross-encoder examples bound input length and tokenize query-paragraph pairs with truncation enabled; the Transformers example reads model logits as scores. These examples establish that the scoring input may be shorter than the original candidate and that scores need not be probabilities. The model table reports retrieval metrics separately from documents processed per second, with runtime measured on a V100 GPU using Transformers v4.

73. TREC 2023 Deep Learning Track

    The passage task distinguishes a passage that merely concerns the topic from one that answers the question: its grade 1 means related but nonrelevant for binary metrics. Document judgments use a different interpretation of grade 1. NIST provides separate judgment transformations for binary measures and nDCG, propagates representative judgments across designated duplicate-passage classes, and derives document relevance from relevant passages. Relevance thresholds, duplicate handling, and the document-versus-passage counting unit therefore belong to the evaluation specification.

74. Cumulated Gain-Based Evaluation of IR Techniques

    Järvelin and Kekäläinen’s 2002 paper argues that binary judgments give equal credit to marginally and highly relevant results. It defines cumulative gain from graded judgments, discounted cumulative gain to reduce the contribution of later ranks, and normalization against an ideal ordering. Its TREC-7 case study shows that system comparisons and statistical conclusions can change with the relevance scale and rank-sensitive measure used.

75. A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track

    Soboroff defines a test collection as documents, topics expressing search needs, and qrels mapping topics to judged-relevant documents; a system turns each topic into a query and returns a ranked run. Pooling judges top results combined from participating systems instead of every corpus document. In the TREC 2006 Terabyte Track, judgments from a deeper random sample ranked systems somewhat differently from the shallow pool, demonstrating that sparse pools can affect evaluation of systems that contributed different candidates.

76. Introduction to Information Retrieval: evaluation of unranked retrieval sets

    For a query, let R be retrieved items and G the relevant items in the evaluation collection. Precision=|R∩G|/|R|=TP/(TP+FP); recall=|R∩G|/|G|=TP/(TP+FN). Recall's denominator includes relevant items that were not retrieved. These are generic set-retrieval definitions, independent of RAGAS. Applying them to chunks or graph evidence requires declaring that counting unit and its relevance judgments; do not mix chunk numerators with document denominators. For a fixed top-k result set, use that set as R. Balanced F1=2PR/(P+R) combines the two when defined.

77. Sentence Transformers InformationRetrievalEvaluator implementation

    The evaluator uses binary relevance sets Rq keyed by query ID and excludes queries without positive judgments. Recall@k=|top_k intersection Rq|/|Rq|. Reciprocal rank is 1/r for the first relevant result at rank r<=k, otherwise zero; MRR@k averages this across queries. DCG@k=sum_i rel_i/log2(i+1), using one-based ranks and rel_i in {0,1}. nDCG divides by the ideal DCG with min(k,|Rq|) leading relevant results. Recall and nDCG are also macro-averaged over included queries. Worked example: two relevant documents at ranks 2 and 4 give Recall@3=0.5, RR@3=0.5 and nDCG@3 approximately 0.387.

78. Introduction to Information Retrieval: Ranked evaluation

    For graded relevance r at rank i, the textbook uses gain 2^r−1 discounted by log2(i+1). DCG@k sums these discounted gains; nDCG divides by the DCG of the ideal descending relevance ordering at the same cutoff. This convention emphasizes high grades, unlike binary relevance or linear-gain variants. For zero ideal gain, the score requires an explicitly stated reporting convention.

79. TREC-COVID Round 5 Relevance Judgments

    CORD-19 releases could drop articles or change their identifiers. TREC-COVID therefore associates qrels with a document release as well as judgment rounds, maps changed identifiers, and excludes dropped documents. NIST explicitly requires using judgments appropriate to the corpus that produced a run. Its qrels distinguish nonrelevant, partially relevant, and fully relevant documents. Most evaluation measures assume unjudged documents are nonrelevant, which is a scoring convention rather than an assessor's judgment.

80. The Text REtrieval Conferences (TRECs)

    Harman describes TREC as a coordinated effort to evaluate retrieval on substantially larger shared collections than those commonly used previously. The first conference occurred in November 1992. Its scale required participating teams to adapt systems, while later rounds explored approaches including passage retrieval and query expansion. The consequential change was shared large-scale experimental infrastructure for comparing independently developed systems, not the invention of relevance evaluation itself.

81. Building Production-Ready RAG Applications

    Establish a task-specific benchmark and evaluate both retrieval components and the complete query-to-answer pipeline.

82. Detecting Good Abandonment in Mobile Search

    A search with no clicked result can reflect either dissatisfaction or successful completion directly from the results page. The paper calls the latter good abandonment and identifies snippets, images, and displayed answers as possible sources of satisfaction. It collects satisfaction reports in a user study and separately obtains crowd judgments for logged mobile searches. Its behavioral analysis includes viewport visibility and gestures rather than treating absence of clicks as a sufficient outcome label.

83. Learning Diverse Rankings with Multi-Armed Bandits

    This 2010 study describes Team-Draft interleaving as combining two rankers into one displayed list while omitting duplicates and tracking which ranker contributed each result. Clicks are credited to the contributing teams to estimate relative preference under shared exposure. Its commercial-search experiment collected 220,000 clicked impressions over two months. Interleaving supports a relative online comparison between two rankings; clicks remain behavior observed under the interleaved presentation, not direct universal relevance labels.

84. Unbiased Learning-to-Rank with Biased Feedback

    A click combines relevance with whether a result was examined. Position changes examination probability, so absent clicks cannot be treated as explicit negative relevance judgments. This paper weights observed feedback by inverse propensity and estimates position effects using randomized rank interventions. The resulting learning objective corrects bias under its observation model and nonzero-support assumptions. It connects ranking decisions to the data later used for training: the deployed ordering partly determines what the system gets to learn.

85. Improving Pinterest Search Relevance Using Large Language Models

    Pinterest uses an expensive joint query–Pin relevance model to supervise a smaller model suitable for live serving. The served model combines reusable embeddings with other ranking features; its relevance score contributes alongside engagement predictions. The paper reports improvements in human-assessed result relevance and separately in search fulfillment, defined through high-significance user actions. Its reported nDCG uses linear relevance gains and an ideal denominator consisting entirely of maximum-grade items, rather than sorting the available judged inventory.

86. User Performance versus Precision Measures for Simple Search Tasks

    Turpin and Scholer studied 30 participants and 50 topics using ranked lists with controlled mean average precision. They measured time to find one relevant document and how many relevant documents users found within five minutes. In this setup, MAP did not predict time to the first relevant document; its relationship with the five-minute collection task was weak. The study demonstrates why a ranking-metric improvement cannot automatically be translated into reduced search effort.

87. Security filters for trimming results — Azure AI Search

    The security-filter pattern stores permitted principal identifiers with documents and matches them against the requesting user's identifiers on each query. Principals may represent users, groups, or roles. Microsoft explicitly states that these filter values are strings, not an authentication or authorization protocol. It also warns that making the identifier field nonretrievable is not field-level security; document restriction depends on applying the security filter to every query.

88. Use a SharePoint indexer to ingest permission metadata — Azure AI Search

    For chunked SharePoint content, ACL fields must be projected onto every searchable chunk; when both parent and child indexes are queryable, both require permission metadata. The preview documentation acknowledges permission-change lag and states that parent-scope permission changes are not automatically picked up on subsequent indexer runs. It also lists unsupported permission-inheritance paths, including enrichment caches and custom Web API skills. Preserving permissions is therefore a path-specific ingestion and synchronization requirement.

89. Zanzibar: Google’s Consistent, Global Authorization System

    Zanzibar demonstrates why content freshness and permission freshness cannot be considered independently. Its example removes a user's access and subsequently adds content; checking the new content against an older ACL could disclose it to the removed user. The system associates content versions with an opaque consistency token and requires later authorization checks to use a sufficiently fresh snapshot. An ACL represents relationships granting users or user sets operations on objects.

90. Controlling access at the document and field level — Elasticsearch

    Elasticsearch document-level security restricts returned documents, but its documentation states that relevance scoring still uses global index statistics. It also warns that some requests can expose aggregate information about inaccessible documents, including terms, field names, and counts. Role definitions combine permissions, and omitting a document query disables that restriction for the corresponding permission entry. Testing visible hits alone therefore does not establish that every search response surface preserves the intended isolation.

91. The refresh parameter — Elasticsearch

    A successful indexing operation need not make its change immediately searchable. With the default refresh behavior, visibility follows later. refresh=wait_for delays the response until a refresh makes the request's changes visible, while refresh=true forces relevant primary and replica shards to refresh. Forced refreshes create small segments that impose indexing, search, and subsequent merge costs. Search visibility is therefore a separate completion boundary with a resource tradeoff.

92. The shard request cache — Elasticsearch

    Elasticsearch's shard request cache invalidates entries when a shard refresh exposes document changes or its mapping changes. It preserves the same near-real-time visibility boundary as uncached search, so extending the refresh interval also extends how long cached entries can remain valid after source changes. By default, this cache stores responses to size-zero requests, including counts, aggregations, and suggestions, rather than returned document hits.

93. Decision record: use versioning to handle race conditions when updating the govuk index

    GOV.UK's accepted design records how publishing events processed out of order can overwrite newer content or recreate an unpublished search result. It selects publishing-event numbers as external versions to reject stale updates. It also explains a deletion gap: removing the record can allow an older publish event to recreate it. A tombstone—a retained record marking the item unpublished—was considered to preserve the newer version but explicitly postponed. The documented consequence was possible exposure of an unpublished page's title and description until correction or reindexing.

94. Changed and Deleted Blobs — Azure AI Search

    Blob indexers detect changes through LastModified, but physical source deletion alone does not remove indexed documents. Configure deletion detection from the first run. Soft deletion leaves an observable marker so the indexer can remove the corresponding search document before permanent source removal. Retention must substantially exceed the indexing interval and allow recovery from failures. Stable document keys and paths are essential to tracking. Restoring a blob may not advance LastModified, so resaving metadata can trigger reindexing. Application implication: verify successful deletion processing before purging markers and retain parent-to-chunk identities for derived cleanup.

95. Aliases — Elasticsearch

    An alias lets an application address an index through a stable name and supports changing its target during reindexing. A remove-and-add action list can atomically switch targets. The documentation separately warns that action lists can partially succeed when an action fails, and describes must_exist=true for failing the entire list on such failure. It also states that alias filters do not apply to document-by-ID retrieval.

96. The Tail at Scale

    Parallel fan-out amplifies occasional slow responses when a request must wait for its component results. Dean and Barroso illustrate how infrequent server outliers become common service-level delays as fan-out increases, and discuss variability from shared resources and queueing. Their hypothetical fan-out example is distinct from the paper's measured service results. Applied to retrieval, a request timeline should identify which branches must finish before merging rather than adding parallel execution durations.

97. Phased Ranking — Vespa

    Vespa separates ranking near stored data from ranking after distributed results are merged. First-phase scoring processes retrieved hits on content nodes; optional second-phase scoring spends more work on a bounded subset. Global-phase scoring operates on merged candidates in the stateless container. The documentation recommends computing data-heavy expressions near their stored vectors and forwarding computed features rather than transferring large vectors for repeated computation. Candidate limits and computation placement are therefore distinct controls over ranking cost.

98. Deadlines — gRPC

    A deadline specifies when a caller will stop waiting; a timeout specifies a permitted duration. gRPC can propagate the remaining budget to downstream calls after subtracting elapsed time. Deadline expiration cancels the RPC, but the server application remains responsible for stopping work it spawned. Applied to distributed retrieval, ending the client wait alone does not establish that retrieval branches or reranking computation have stopped.

99. Vector Search Benchmark[eting]

    Benchmark the application's own workload before relying on published performance comparisons.

100. OpenTelemetry: Traces

     A distributed trace connects operations through spans carrying identity, timing, attributes and events. Parent-child relationships and links represent related work across boundaries. Applied to an agent, model requests, tool dispatch and downstream calls can be correlated so a failed run can be reconstructed from observations. A span’s error-free status describes that instrumented operation; it does not by itself establish that the requested business outcome was correct.

101. Data is Your Differentiator: Building Secure and Tailored AI Systems

     A retrieval application should log user queries, retrieval hits, and model responses before even a pilot, because these traces are necessary for troubleshooting and iterative improvement.

102. Navigating RAG Optimization with an Evaluation-Driven Compass

     Use variation in aggregate metrics to select failing examples, then inspect their retrieved documents before choosing the next intervention.

103. Building Production-Ready RAG Applications

     Separate irrelevant retrieved context from missing required evidence; increasing top-K addresses neither problem universally.

104. NIST randomized blocks: applying controlled comparisons to RAG evidence

     NIST describes holding nuisance factors constant within blocks and randomizing remaining variation. Proposed RAG application: for each fixed query, compare the recorded baseline with a candidate-boundary intervention replacing candidates with independently verified sufficient source passages, then run the unchanged reranker, assembler and generator. Separately replace only the final evidence block with sufficient passages, bypassing retrieval and assembly. Keep corpus/ACL/time snapshot, question, prompt template, model/version, decoding settings, token budgets and evaluator fixed; record passage identities, order and every resulting context. For assembly diagnosis, replay the same candidates through original versus evidence-preserving assembly. Match evidence length and position where feasible; otherwise the treatment changes those too. Compare boundary coverage and answer support, not answer wording alone.

105. Data is Your Differentiator: Building Secure and Tailored AI Systems

     Cache lookup should consider semantically similar questions rather than require exact text matches, allowing prior results to bypass another foundation-model invocation.

106. Data is Your Differentiator: Building Secure and Tailored AI Systems

     Evaluation findings should feed an iterative update-and-retest loop: refresh stale data or revise retrieval strategy, then run automated evaluations through a maintained test suite.

107. What We Learned from Using LLMs in Pinterest

     The speakers reported international relevance and search-engagement gains despite predominantly US human training data, using one multilingual model across languages.