Recommendation Systems: Matching People With Useful Choices

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. How YouTube recommendations work

   YouTube documents different contextual emphasis across recommendation surfaces: the currently watched video is the main signal for the next-video suggestion, while homepage recommendations primarily use watch history. Its listed inputs include deliberate judgments such as likes, dislikes and satisfaction surveys alongside behavioral histories. Surveys ask viewers to rate watched videos, providing satisfaction information beyond watch time. The service also accepts Not interested and Don’t recommend channel feedback as recommendation signals.

2. How Zalando Uses Technology to Help Customers Find the Right Size

   Zalando’s June 2026 account describes size recommendations intended to help customers choose garments that fit and reduce avoidable returns. Inputs include brand information, purchase and return behavior, customer fit feedback and assessments from specialists who try on products. Product-level flags indicate unusually small or large sizing; personalized recommendations also use a customer’s usual sizes and fit preferences. Customers can identify familiar products, including purchases elsewhere, to improve filtering and selection. This provides a concrete recommendation workflow whose usefulness is assessed beyond the initial purchase or click.

3. The effects of a personalized recommendation system on students’ high-stakes achievement scores: A field experiment

   Math Nation recommended algebra videos using mini-assessment results, selecting the topic with the largest predicted assessment-score improvement. Students could ignore the recommendation. The comparison policy followed a predetermined curriculum sequence. Evaluation used the end-of-course algebra assessment, separating the recommendation’s immediate optimization target from its downstream educational goal. The study assigned 16,406 students, but excluded 5,448 from analysis because of missing covariates. Its initial analysis did not find a significant treatment coefficient; subsequent exploratory analyses examined differences by usage level.

4. Recommendation stages: eligibility, scoring and list constraints

   Candidate generation retrieves a manageable subset; scoring estimates item-level desirability; re-ranking applies additional constraints such as explicit dislikes, diversity and freshness. Teaching distinction inferred from these operations: eligibility is a Boolean decision about whether an item may appear, preference scoring compares eligible items, and a list objective evaluates their combined selection or ordering. Removing a disliked item is an eligibility constraint, whereas boosting freshness changes relative priority. Diversity depends on which other items appear, so independently sorting item scores need not optimize it. Multiple generators can contribute candidates; later ranking cannot select an item absent from that candidate set.

5. Off-policy evaluation for slate recommendation

   A slate is an ordered collection of recommended items. For a slate-level reward, standard inverse-propensity evaluation weights each logged reward by the target policy’s probability of that slate divided by the logging policy’s probability of it, conditional on context. These are selection probabilities, not click probabilities. The logging policy must support every slate the target can choose. Because possible lists grow combinatorially, full-slate weighting can have prohibitive variance. The paper’s alternative estimator reduces data requirements by assuming expected reward decomposes into context-dependent item-position contributions.

6. Personalization in the Era of LLMs

   Candidate generation reduces the item pool before one or more rankers select the final recommendations.

7. Two-tower retrieval, approximation and compatible deployment artifacts

   Separate user/query and item models produce embeddings in a shared scoring space. Item vectors can be computed before requests and indexed with item identifiers; serving computes the query vector and retrieves high-scoring candidates. Brute force evaluates the catalog, while ScaNN searches approximately and may miss exact top-K items. Search breadth and rescoring settings trade latency against retrieval accuracy; compare approximate results with brute force on the same candidates. The tutorial exports a SavedModel containing the query model and index. Engineering inference: deployment must keep query weights/preprocessing compatible with candidate vectors, identifiers, scoring conventions and index. Changing the item encoder requires recomputing affected vectors and updating or rebuilding the index; changing both towers requires a matching index. Retain a compatible bundle for rollback.

8. GroupLens: An Open Architecture for Collaborative Filtering of Netnews

   Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom and John Riedl presented GroupLens in October 1994. The MIT–Minnesota system addressed information overload in Usenet discussion groups, where readers interested in the same topic could still value different articles. Reader applications collected ratings and displayed predictions; interoperable rating servers predicted scores from agreement among readers’ previous ratings. This extended topic subscription and keyword filtering with judgments from other people. The paper also recognized an evaluation problem: ratings collected after one recommender selects articles can unfairly evaluate another recommender that would have selected different articles.

9. Amazon.com Recommendations: Item-to-Item Collaborative Filtering

   Greg Linden, Brent Smith and Jeremy York’s January–February 2003 report describes Amazon’s deployed response to expensive customer-neighborhood computation. It constructs an item-to-item similarity table offline from shared purchasing behavior. At request time, the system looks up items related to a customer’s purchases or ratings, combines those results and selects recommendations. The reported online computation depends on the customer’s history rather than scanning the entire customer population and catalog. The shopping-cart example uses the current cart as recommendation context, showing that personalization need not depend only on a durable profile.

10. Matrix Factorization Techniques for Recommender Systems

    Yehuda Koren, Robert Bell and Chris Volinsky’s August 2009 article explains the role of latent-factor models in their Netflix Prize work. Rather than relying only on neighboring users or items, these models learn compact user–item compatibility factors and incorporate user and item biases, implicit feedback and temporal changes. Their reported comparisons show lower rating-prediction error as these components are added. Their progress-prize ensembles contained more than 100 predictors, mostly factorization variants, illustrating that successful entries combined models rather than identifying one universally best recommender.

11. The Netflix Prize

    Netflix organizers James Bennett and Stan Lanning describe the competition launched in October 2006. Netflix released more than 100 million dated, one-to-five-star ratings and challenged entrants to reduce Cinematch’s rating-prediction RMSE by another 10%. Predictions for withheld ratings were divided between a leaderboard subset and an undisclosed final-test subset. Recent ratings were selected to reflect predicting future opinions from earlier ratings. The competition therefore standardized a particular prediction problem; its improvement target was not a percentage increase in viewing satisfaction or recommendation usefulness.

12. YouTube training examples, sampled classes and watch-time ranking

    The 2016 candidate model predicts a watched video from prior context, using watches beyond recommendation impressions and a fixed number of examples per user. It samples negative video classes from a background distribution and applies importance weighting to correct the sampling. These sampled alternatives are not observed dislikes. To prevent future leakage, a selected watch becomes the label and only preceding actions enter its history features. Ranking uses clicked and unclicked impressions; clicked examples receive watch-time weights and unclicked examples unit weight. Consequently, learned odds approximate expected watch time when click probability is small, rather than representing ordinary click probability.

13. Foundation Model for Personalized Recommendation

    Ko-Jen Hsiao, Yesu Feng and Sudarshan Lamkhede’s March 2025 account motivates shared preference learning by the maintenance burden of separately trained recommendation models. A common model learns from interaction histories and content, supplying representations to downstream applications. Its new-title mechanism combines metadata-based representations with learned item-ID representations using an age-conditioned mixing layer. Newly introduced titles rely more on metadata; established titles can rely more on interaction-derived identity information. This addresses item cold start—estimating appeal before relevant interaction history exists—while allowing behavior to distinguish titles whose descriptions look similar.

14. Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs

    The Indeed example starts with expert-labeled user–job pairs and treats precision of rejection as a critical constraint.

15. Preparing item interaction data for training — Amazon Personalize

    Amazon Personalize supports interaction context such as device type and accepts context when recommendations are requested. Its impression example distinguishes returned recommendations from displayed items: an application removes out-of-stock shoes, then records the displayed subset with the clicked item. Alternatively, a recommendationId links an interaction to an earlier recommendation response. Explicit impression lists disregard item order. For supported configurations, impressions guide exploration during recommendation requests rather than model training.

16. Modeling delayed conversion feedback

    Let C denote eventual conversion, D conversion delay, E elapsed observation time and Y whether conversion has already been observed. Y=0 can mean C=0 or a future conversion with D>E. If p(x)=P(C=1|x) and F(e|x,C=1) is the delay CDF, then P(Y=0|x,E=e)=1-p(x)*F(e|x,C=1). Thus an immature observation is censored evidence, not a confirmed negative. Chapelle jointly fits conversion probability and a feature-dependent exponential delay model, assuming (C,D) is independent of E conditional on X. The study uses a 30-day attribution window and last-click attribution; later conversions are outside its target.

17. One model to rule recommendations: Netflix's Big Bet

    Treat an interaction token as a structured event and choose its granularity together with the available context window.

18. Point-in-time historical feature joins

    Feast historical retrieval starts with entity keys and an event timestamp for every training row. It joins historical feature values by scanning backward from that row's timestamp, limited by the feature view's time-to-live window. TTL is relative to each historical event, not the date the retrieval job runs. A row with no eligible prior feature observation remains unmatched rather than receiving a future value. This reconstructs timestamped historical feature state instead of joining today's latest values. Engineering limitation: event-time filtering alone cannot prove that late-arriving or retrospectively recomputed data was actually available then; feature timestamps and availability semantics must reflect that requirement.

19. Content-based Filtering

    Content-based recommendation represents item properties and user preferences in a shared feature space and scores candidate items by similarity, such as a dot product. It can reason about an item through its attributes rather than requiring that item to already have a dense interaction history. This distinguishes metadata-based matching from collaborative factorization. The representation still determines which interests can be expressed, and a new user still needs preference evidence or an explicitly chosen initial policy.

20. Item-based Collaborative Filtering Recommendation Algorithms — Memory-based Collaborative Filtering Algorithms

    User-based collaborative filtering finds neighbors whose historical ratings or purchased-item sets resemble the target user’s. It combines those neighbors’ preferences to predict an item’s suitability or produce a recommendation list. The neighborhood is a relationship inferred from recorded behavior, rather than a requirement that users explicitly identify one another or describe item content.

21. Item-based Collaborative Filtering Recommendation Algorithms — Item-based Collaborative Filtering Algorithm

    The item-based method considers items already rated by the target user, identifies those most similar to the item being scored, and predicts its rating through a similarity-weighted combination of the user’s ratings on those neighbors. This differs from selecting neighboring people: shared item relationships connect the person’s existing ratings to a new item.

22. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    Shared purchases define a local peer group, whose purchases supply recommendations for items the target customer has not bought.

23. Google Recommendation Systems: Matrix factorization

    Matrix factorization learns compact user and item embeddings whose dot products approximate observed feedback. The interaction matrix is sparse: missing entries do not directly say that a user dislikes an item. Fitting only positive observations can produce a trivial predictor, while treating all missing entries equally as negatives can let them dominate. Weighted objectives balance observed and unobserved interactions and can adjust for highly frequent users or items. Representation learning and the meaning assigned to feedback are therefore inseparable.

24. Deep Learning, Chapter 2: Linear Algebra

    For real vectors, the dot product sums corresponding coordinate products; L2 length is sqrt(sum(x_i²)), and Euclidean separation is ||x-y||₂. The identity x·y=||x||₂||y||₂cos(theta) separates magnitude from direction. For nonzero vectors, dividing by length produces unit vectors whose dot product equals cosine; expanding squared distance gives ||u-v||₂²=2-2u·v, hence equivalent rankings. Orthogonal transformations preserve these inner products and distances because QᵀQ=I. Constructed example: query (1,0) gives candidates (2,2) and (1,0) dot products 2 and 1 but cosines 1/sqrt(2) and 1. For directions 0°, 45°, and 90°, one-minus-cosine violates triangle inequality: 1>2-sqrt(2).

25. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    The workshop's GraphRAG design combines vector search with knowledge graph traversal and graph data science, using purchase behavior to personalize retrieved products.

26. Anonymous session intent from ordered interactions

    Session-based recommendation can operate without a persistent user identifier. An ordered click sequence updates a recurrent hidden state, schematically h_t=g(W*x_t+U*h_(t-1)), from which the model scores the next item. The current recommendation therefore depends on earlier session actions rather than only the last click. In the paper's session-parallel training, the next event supplies the target and hidden state resets when a session ends. This assumes separate sessions are independent. Session evidence describes the current activity; it does not establish enduring preferences across visits. The paper compares global popularity, within-session popularity and item-neighbor baselines, with within-session repetition particularly relevant to session popularity.

27. Persistent preference, temporal drift and transient effects

    Koren separates stable user bias, gradual drift and day-specific effects: b_u(t)=b_u+alpha_u*dev_u(t)+b_(u,t). Item biases can vary by time bin, and user latent factors can also change over time. These components distinguish persistent patterns from transient behavior while learning across the historical record. A temporal neighborhood model weights an earlier interaction by exp(-beta_u*|t-t_j|), with a learned user-specific decay rate. This makes recency weighting concrete without requiring old examples to be discarded during training. In the reported Netflix experiments, globally downweighting older training instances performed best with no decay; older observations retained useful collaborative structure.

28. Personalization in the Era of LLMs: user steering

    The speaker describes three controls: spoken requests to AI DJ, natural-language prompts for generated playlists, and edits to an exposed textual taste profile. Prompts state the desired selection for a request. Profile edits supply explicit corrections to inferred preferences, such as wanting more of an artist or disliking a particular recommended podcast. He says those edits return to the generative model to influence what it recommends; this is described as recommendation input, not merely a changed explanation. Separately, he describes projecting a user embedding into the LLM space and inserting it into the prompt as a soft token. The transcript does not establish that profile edits update that embedding or model weights, nor provide a controlled before/after recommendation result.

29. 360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI

    Longer history improved performance in the reported experiment until performance dropped at the longest contexts.

30. Re-identification risk in released recommendation histories

    Removing direct identifiers does not necessarily anonymize sparse behavioral records. Narayanan and Shmatikov show how auxiliary knowledge of a few movie ratings and approximate dates can help link a person to a released Netflix rating record. Sparse combinations can be distinctive even when individual movies are common. A successful linkage can expose additional ratings that were not public in the auxiliary source. The paper includes experiments with imperfect auxiliary information and a small IMDb-based proof of concept, demonstrating a concrete risk from publishing detailed recommendation histories.

31. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    Use separate personalized-search and history-based recommendation retrievers, then supply both product lists to the generation prompt.

32. Relevance scoring in hybrid search using Reciprocal Rank Fusion — Azure AI Search

    Hybrid retrieval can combine independently ranked lexical and vector results. Reciprocal Rank Fusion assigns a document a contribution of 1/(rank + k) in each list where it appears and sums those contributions. It merges ranks rather than directly comparing BM25 and vector score magnitudes. The fusion constant k is different from a nearest-neighbor result count. Azure's semantic reranker runs after fusion and reports a separate score. Fusion, candidate count, and reranking are therefore distinct controls with different effects on the evidence set.

33. Filtering recommendations and user segments — Amazon Personalize

    Amazon Personalize applies request-time filters for criteria such as excluding previously purchased products or restricting items by age group. Promotions apply additional business rules to a configurable subset of recommendations. Filter state updates asynchronously: the documentation describes updates within seconds for PutEvents and PutActionInteractions, and within 20 minutes of the last import for other imported data. A configured filtering rule therefore has a separate data-propagation boundary.

34. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    An optional graph match can preserve semantic-search results when the behavioral graph supplies no match.

35. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    Graph-first filtering is a proposed scaling option, but sparse local graphs can over-restrict semantic search.

36. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    The score product is acknowledged as an expedient baseline; production choices should be evaluated against labeled recommendation outcomes.

37. Personalization in the Era of LLMs

    The described transformer approach combines user interaction history with request context and the item being recommended.

38. Pointwise, pairwise and listwise learning to rank

    Pointwise training uses individual query-item examples with relevance labels and a regression or classification loss, such as squared error or binary log loss. Pairwise training uses two items for the same query plus their preferred order. For a preferred item i over j, RankNet's binary cross-entropy reduces to -log(sigmoid(s_i-s_j)); increasing the preferred item's score relative to the other reduces loss. Listwise training groups a query's candidate list and judgments. ListMLE minimizes -log P(observed ordering|scores), assigning probability to the whole ordering; other listwise methods optimize ranking-metric surrogates. These categories describe training units and objectives, not necessarily different serving architectures.

39. BPR: Bayesian Personalized Ranking from Implicit Feedback

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner and Lars Schmidt-Thieme’s 2009 BPR paper makes personalized ordering the training objective instead of reconstructing individual feedback entries. Training triples contain a user, an interacted-with item and an unobserved alternative; the model is trained to prefer the former. The same criterion applies to matrix-factorization and adaptive item-neighborhood models. In the authors’ Rossmann purchase and Netflix interaction experiments, BPR improved AUC against the studied training alternatives, demonstrating that changing the objective can matter without replacing the model family.

40. Logistic regression: features, parameters and conditional probabilities

    For numeric features x, fitted weights w and intercept b, logistic regression computes z=b+sum_j(w_j*x_j), then p=1/(1+exp(-z)). The linear score z is the log odds log(p/(1-p)); p lies strictly between zero and one for finite z. In a click-label application, p estimates P(click=1|x) under the modeled data distribution. Features are observations supplied to the model; weights and intercept are parameters learned from labeled examples. A positive weight increases log odds when that feature increases while other features remain fixed. Interpreting this as user utility would require an additional definition and evidence: the modeled outcome here is a click.

41. Evaluate correctness, calibration, robustness and efficiency separately

    HELM evaluates models under standardized adaptation conditions across scenarios and separate metrics. Classification accuracy is the fraction correct; other tasks require explicit metrics such as answer overlap or retrieval ranking quality. Calibration asks whether predictions assigned confidence c are correct approximately c of the time. Binned ECE averages |bin accuracy minus bin confidence| weighted by bin size; HELM uses ten equal-count bins. Robustness tests behavior under specified input perturbations rather than assuming clean accuracy transfers. Efficiency is a separate resource measurement: HELM distinguishes observed serving time, denoised runtime and runtime under standardized hardware/software, accounting for prompt and output lengths. Report the evaluation data, prompting, decoding, scoring and resource conditions alongside results. HELM warns that its limited example samples and three prompt-selection seeds do not establish statistical significance for every comparison.

42. Learned Ranking Function: From Short-term Behavior Predictions to Long-term User Satisfaction

    YouTube’s reported Learned Ranking Function combines predicted user behaviors with user context and video features, while constraining secondary objectives. It explicitly models value after abandoning a recommendation slate because users may continue elsewhere on the platform. In an online ablation that removed this abandonment-value baseline, the watch-page-attributed metric increased while the platform’s primary metric decreased. This supplies a concrete case in which improving outcomes attributed to one recommendation surface conflicts with the broader product objective.

43. What We Learned from Using LLMs in Pinterest

    The distilled student scores relevance late in the search pipeline, after candidate generation and early ranking and immediately before blending.

44. Temporal protocols and distinct recommendation-quality dimensions

    Temporal evaluation can step through events, predicting each before making it available to subsequent predictions; a cutoff protocol hides later observations. Rating error, usage prediction and ranking accuracy answer different questions. Catalog coverage can measure distinct recommended items divided by catalog size over a defined experiment. Intra-list diversity can average pairwise item distances, requiring a specified similarity representation. Novelty concerns items previously unknown to the user; removing recorded history cannot establish novelty when history is incomplete. Statistical uncertainty concerns variation in estimated performance, whereas prediction confidence concerns an individual prediction. Comparisons should respect dependence among observations from one user and separate tuning from final testing.

45. What Is Serendipity? An Interview Study to Conceptualize Experienced Serendipity in Recommender Systems

    Through interviews with 17 frequent recommender users in Belgium, the authors develop a framework in which serendipitous encounters feel fortuitous, refreshing and enriching. Novelty concerns the user's familiarity with content; unfamiliar content can still match established tastes. Refreshing discovery can also reconnect someone with a forgotten interest, so previously consumed content is not automatically excluded. Participants described appreciating unfamiliar series in a favorite genre and rediscovering music they already knew. The framework requires perceived value, not merely unusual content. It therefore concerns a person's experience rather than differences among items in a displayed list; the latter distinction is an explanatory inference, not an experimentally tested comparison.

46. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries

    Maximal Marginal Relevance constructs a list incrementally by balancing relevance to a query or user profile against similarity to items already selected. A candidate’s contribution therefore depends on the developing list, not only its standalone relevance. A tunable parameter controls the balance: one endpoint preserves relevance ordering, while the other emphasizes diversity among retrieved candidates. The authors distinguish broad exploration of a topic from deliberately reading overlapping material in depth.

47. Learning a Personalized Homepage

    Chris Alvino and Justin Basilico’s April 2015 account treats Netflix’s homepage as a joint choice of rows, titles and placement. Independently ranking rows can produce many redundant variants of one interest. A stage-wise alternative scores each next row against rows and videos already selected; looking further ahead increases computation and still need not find the optimal page. Device limits, filtering and deduplication constrain selection. The authors also distinguish freshness from stability: recommendations should respond to actions without making recently suggested titles difficult to find again.

48. Provider exposure as a ranking objective

    User relevance describes an item's usefulness to the requester; provider exposure describes the attention opportunity allocated to its supplier. For probabilistic rankings, let P_ij be the probability item i occupies position j and v_j the attention weight of that position. Expected exposure is E_i=sum_j P_ij*v_j. Small relevance differences can cause large exposure differences when higher ranks receive much more attention. Singh and Joachims maximize expected ranking utility subject to explicit exposure constraints, including equal average group exposure or exposure proportional to group relevance. Those constraints can change the utility-maximizing ordering.

49. Unbiased Learning-to-Rank with Biased Feedback

    A click combines relevance with whether a result was examined. Position changes examination probability, so absent clicks cannot be treated as explicit negative relevance judgments. This paper weights observed feedback by inverse propensity and estimates position effects using randomized rank interventions. The resulting learning objective corrects bias under its observation model and nonzero-support assumptions. It connects ranking decisions to the data later used for training: the deployed ordering partly determines what the system gets to learn.

50. Recommendations as Treatments: Debiasing Learning and Evaluation

    Observed feedback is selected by the process that decides which user-item pairs are exposed or rated. Averaging error only over those observations can favor a model that matches the selection bias. Inverse-propensity scoring weights an observed contribution by the inverse probability of its observation, correcting this bias under the paper’s assumptions. Very small propensities produce large weights and greater variability; estimating propensities adds another modeling problem. This makes coverage and uncertainty essential parts of counterfactual evaluation.

51. Algorithmic confounding and simulated homogenization

    The study simulates users with fixed latent preferences who choose items using both their own imperfect utility information and recommendation rank. Recommenders repeatedly train on the resulting behavior without separating recommendation influence from underlying preference. Under these modeled conditions, repeated training increases behavioral similarity beyond that needed for ideal utility. Homogenization is measured by Jaccard overlap of users' consumed-item sets; item-consumption inequality is measured separately using a Gini coefficient. Similar homogenization can coexist with different consumption distributions, so it should not be equated automatically with concentration on a few global hits.

52. Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms

    Recommendation logs contain the outcome of the item actually displayed, not the outcome of every alternative. This paper’s replay evaluator retains a logged event only when the policy being evaluated chooses the same action as the logging policy. Under its independent-event and uniformly randomized-logging assumptions, the retained history matches the distribution of an online run. The method makes the missing-counterfactual problem explicit and shows why logging policy matters to evaluation, not just model training.

53. Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix

    Aridor, Chou, Kallus, Scheid, Tran and Zielnicki’s August 2026 preprint reports a 60-day Netflix holdback experiment conducted from February to April 2025. Controls retained the previous recommendation system while treatment received twelve algorithmic updates without interface changes. Recommendations and plays shifted from the most popular titles toward moderately popular titles, with little change for the least-popular half of titles. Overall viewing and distinct titles played increased, while aggregate play concentration decreased. The study demonstrates why exposure distribution and total product activity should be measured separately: a policy can change both how much people consume and which inventory receives attention.

54. Strategic manipulation through fabricated rating profiles

    Shilling attacks inject fabricated user profiles to promote or suppress target items. Profiles combine deliberately extreme target ratings with filler ratings intended to resemble ordinary users, influencing collaborative similarities and predictions. The paper studies random and average-rating filler strategies and finds that effects depend on the recommender algorithm and whether success is measured by prediction changes or recommendation-list occupancy. The defining difference from ordinary noisy feedback is strategic construction toward a target outcome: attackers choose correlated ratings and identities to influence other users' recommendations rather than report independent experiences.

55. Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs

    Hash-based item IDs provide no content signal, leaving new and tail items dependent on insufficient interaction evidence.

56. Learning new-user preferences through informative rating requests

    A recommender can ask a new user to rate selected items. Selection by rating frequency increases the chance the user knows an item; selection by rating entropy H=-sum_r p_r*log(p_r) favors items on which opinions differ. Entropy alone can select obscure items the user cannot rate, motivating strategies that balance familiarity and informativeness. Adaptive strategies use answers already supplied to choose later questions. These methods require existing population feedback plus the new user's responses. In contrast, new-item content matching requires item attributes and a usable representation of the user's interests; metadata supplies evidence about an item, not an otherwise unknown user's preferences.

57. Rules of ML: baselines, feature consistency and freshness monitoring

    Google recommends simple heuristics before sufficient training data exists, including install counts, install rates and human editing. Such policies provide executable baselines; their adequacy must be measured. Hard exclusions can remain outside the learned model. Training-serving skew can arise from different feature pipelines, changing data or feedback loops. Log actual serving-time features for at least a sample of requests and compare them with training computations. Compare training, holdout, next-day and live performance separately; the same example should produce the same prediction through equivalent pipelines. Monitor model-age effects and quantify undesirable recommendations. Engineering inference: successful HTTP responses cannot establish feature consistency or recommendation quality.

58. Contextual bandits: reward estimation plus exploration

    A contextual bandit observes request context, chooses one available action and receives reward only for that action. LinUCB selects argmax_a[x_a·theta_hat_a+alpha*sqrt(x_a^T*A_a^-1*x_a)]. The first term estimates reward; the second rewards uncertainty under the linear model, encouraging exploration of insufficiently observed actions. A_a summarizes observed feature information plus regularization. After observing the selected action's reward, the algorithm updates its estimates. Unlike fixed-policy user assignment in an ordinary A/B test, this policy adapts action selection to context and accumulated feedback while seeking cumulative reward.

59. On the Likelihood That One Unknown Probability Exceeds Another in View of the Evidence of Two Samples

    William R. Thompson’s 1933 paper, written at Yale’s Department of Pathology, addresses acting while evidence is still accumulating. It proposes allocating treatments according to the current probability that one is better, including allocation directly proportional to that probability. This avoids immediately committing everyone to an uncertain winner while continuing to collect information. The mathematical development compares two unknown event probabilities using observed successes and failures and independent uniform prior assumptions.

60. A Tutorial on Thompson Sampling

    Russo, Van Roy, Kazerouni, Osband and Wen’s 2018 tutorial distinguishes exploitation of current estimates from exploration that gathers information. In its binary-outcome example, Thompson sampling maintains a distribution over each action’s unknown success probability, samples one plausible probability per action and selects the largest. Only the chosen action’s observed outcome updates its distribution. This differs from epsilon-greedy’s fixed allocation to uniformly random exploration. The tutorial also describes restricting choices to admissible actions and explains why changing environments require continued learning rather than permanent certainty.

61. Contextual Bandit Content Personalization — VowpalWabbit 9.1.0

    The published simulation gives the same fictional visitor different preferred article categories in the morning and afternoon. Each learning record contains context, the chosen article, its selection probability and observed reward. The policy returns a distribution over actions; the example distribution [0.7, 0.1, 0.1, 0.1] selects the first action with probability 0.7. Sampling returns both the action and its probability for the later learning record. User–action feature interactions allow preferences to depend jointly on identity, time and article category.

62. Guardrails and interference in online experiments

    Guardrail metrics represent outcomes experimenters seek not to degrade, distinct from the primary success metric. Standard treatment-effect analysis assumes a unit's outcome does not depend on other units' treatment assignments. Social connections, competition for shared marketplace resources or shared infrastructure failures can violate that assumption. Then a treatment-control difference need not estimate the effect of deploying one policy to everyone. Cluster randomization groups interacting units, while switchback designs alternate treatment over time; both require design-specific analysis. Clustering can reduce interference while also reducing effective sample size and statistical power.

63. Leveraging Unlabeled Data to Scale Blocking for Record Linkage

    Blocking selects candidate record pairs before expensive matching. Restating the paper's coverage and precision definitions: let U contain eligible pairs, T⊆U the true matching pairs, and C⊆U the distinct candidates. Candidate recall is |C∩T|/|T|; candidate precision is |C∩T|/|C|. Reduction ratio is 1−|C|/|U| and measures avoided comparisons, not matching accuracy. For deduplication of n records, U has n(n−1)/2 unordered non-self pairs; cross-dataset linkage uses its permitted cross-product. Evaluate blocking against labeled truth that includes matches outside C. Measuring only scored candidates cannot reveal those excluded matches. The paper separates training, development, and test data when learning and evaluating blocking rules.

64. Splink evaluation tutorial: labeled pairs and threshold selection

    Splink evaluates pair predictions against labeled matches and nonmatches, reports false positives and false negatives, and varies the acceptance threshold to examine precision and recall. It requires representative labels for unbiased accuracy analysis. Evaluation decomposition inferred from these definitions: let T be all true matching pairs, C the candidates, and A⊆C the accepted pairs. Decision precision is |A∩T|/|A|. Conditional decision recall is |A∩T|/|C∩T|; end-to-end pair recall is |A∩T|/|T|. Thus end-to-end recall equals candidate recall times conditional decision recall when denominators are nonzero. End-to-end false negatives include both T\C and (T∩C)\A. Report conditional and end-to-end results separately; labels sampled only from high-scoring candidates cannot establish population recall.

65. Recall@K: relevant-item denominator

    For a query or user with relevant-item set R and recommended top-K set T_K, Recall@K=|T_K intersect R|/|R|. The denominator is the number of relevant items in the evaluation universe, not K. TensorFlow Ranking binarizes graded labels at y>=1 and sorts predictions with randomly broken ties. With one held-out relevant item, recall is one if it appears in the top K and zero otherwise. With several relevant items, retrieving one is only partial recall. A dataset result aggregates these per-list values according to an explicit weighting rule.

66. NDCG@K: gains, discounts and ideal normalization

    For relevance labels y_(r) in predicted rank order, DCG@K=sum_(r=1..K) g(y_(r))*d(r). A standard choice uses gain g(y)=2^y-1 and discount d(r)=1/log2(r+1), so more relevant items contribute more and lower positions contribute less. IDCG@K computes the same sum after sorting the same candidate labels into their ideal relevance order. NDCG@K=DCG@K/IDCG@K; with nonnegative gains and positive ideal gain, it ranges from zero to one. TensorFlow Ranking exposes gain and discount functions and uses pow_minus_1 and log2_inverse by default.

67. Evaluation metrics under sampled candidates

    Ranking one relevant item against sampled nonrelevant items changes its rank distribution. With full-catalog size n, true rank r and m uniformly sampled negatives with replacement, sampled rank is 1+Binomial(m,(r-1)/(n-1)). Consequently, expected sampled Recall@K is the binomial probability of at most K-1 sampled items outranking the relevant item, not the full-catalog indicator r<=K. The paper shows that averaging these nonlinear transformations can reverse model comparisons even in expectation. Very small samples make distinct ranking metrics increasingly resemble AUC. Uniform sampling without replacement instead gives a hypergeometric rank distribution.

68. Randomized policy comparison and experimental uncertainty

    Randomly assigning eligible experimental units to control and treatment supports causal comparison of policy variants. For a scalar outcome, estimate the absolute effect as mean(Y_treatment)-mean(Y_control). Under independent units and appropriate large-sample conditions, an approximate 95% confidence interval is the estimated difference plus or minus 1.96 standard errors. Power depends on outcome variation, sample size and the effect one wants to detect. Choose an overall evaluation criterion before interpreting results, and examine operational metrics as well. The experiment measures outcomes during its observation period; short tests do not automatically identify long-term effects.

69. Controlled experiments on the web: survey and practical guide

    Section 3.6 distinguishes temporary inefficiency while established users learn changed navigation from extra investigative clicking prompted by a new feature. The authors recommend running some experiments for multiple weeks and separately analyzing new users. They also recommend considering repeat visits, abandonment and delayed conversions when defining the overall evaluation criterion. These are methodological recommendations, not reported longitudinal observations demonstrating that a particular recommendation benefit persists.

70. One model to rule recommendations: Netflix's Big Bet

    Centralizing representation learning can reduce duplicated feature work while retaining thinner application-specific models.

71. Integrating Netflix’s Foundation Model into Personalization Applications

    In November 2025, Divya Gadde, Ko-Jen Hsiao, Dhaval Patel and Moumita Bhattacharya reported three production integration paths: consuming embeddings, incorporating a model subgraph and fine-tuning the model for applications. Different latency requirements and technical constraints motivated coexistence. Their embedding workflow retrains monthly, fine-tunes daily while adding new title IDs, then runs batch inference and publishes versioned, timestamped representations. Retraining and daily updates can shift the representation space, requiring stabilization for downstream consumers. Batch representations reduce integration costs but can become stale before a recommendation request.

72. Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU

    Remove personalized prompt fields and precompute video-to-video recommendations offline to avoid online model inference for those candidates.

73. Netflix Recommendations: Beyond the 5 stars (Part 1)

    In April 2012, Xavier Amatriain and Justin Basilico reported that Netflix put matrix-factorization and restricted-Boltzmann-machine components from the 2007 progress-prize work into production after adapting them for scale and incoming ratings. Netflix did not deploy the final 2009 grand-prize ensemble: its additional offline accuracy did not justify the engineering effort, and the product problem had changed. Streaming, launched in 2007, shifted selection from filling a DVD queue days ahead to finding something to watch immediately. It also supplied viewing feedback unavailable during mailed-DVD consumption. The account connects algorithm selection to changing workflow, available observations and deployment costs.

74. Netflix architecture: online, nearline and fallback computation

    Netflix describes online computation for current request context, offline computation for expensive batch results, and event-triggered nearline computation that asynchronously stores updated results between requests. Offline computation offers flexibility but may become stale. Online computation has latency and availability constraints and depends on online data sources; the article recommends a fast fallback such as a precomputed result when those constraints cannot be met. Periodic aggregation can produce popularity baselines. Together these mechanisms trade computation, freshness and response reliability rather than requiring every recommendation to be computed from scratch synchronously.

75. Turning Fails into Features: Zapier’s Hard-Won Eval Lessons

    Use a limited live A/B test to assess whether a model or prompt change improves user outcomes.

76. One model to rule recommendations: Netflix's Big Bet

    Multi-step generation offers a place to incorporate collection-level diversity and business rules, with sparse behavioral rewards potentially guiding the process.

77. One model to rule recommendations: Netflix's Big Bet

    A reusable user representation needs stability as interaction history changes, not just predictive quality for the next event.

78. Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs

    Netflix's unified contextual ranker uses a common input schema and fills missing task-specific fields, including substituting an item title when no search query exists.

79. Personalization in the Era of LLMs

    The described system projects a user embedding into the LLM's representation space and inserts it into the prompt as a soft token.

80. One model to rule recommendations: Netflix's Big Bet

    Serving latency and cost can limit useful model scaling even while training improvements continue.

81. Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU

    Serving cost and latency can make an otherwise useful recommender impractical, requiring smaller checkpoints and substantial inference optimization.