Data Quality and Curation

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. The Government Data Quality Framework

   Data quality means suitability for an intended purpose, with requirements determined by users and their needs. The framework distinguishes completeness, uniqueness, consistency, timeliness, validity and accuracy; complete or correctly formatted records can still contain incorrect values. Different uses can require different priorities and tradeoffs. Quality assessment belongs throughout collection, processing, sharing and maintenance, with documented limitations, accountable owners and communication of changes. The framework explicitly distinguishes input-data quality from assurance of analytical outputs.

2. Datasheets for Datasets

   Datasheets proposes documenting purpose, composition, collection, preprocessing, uses, distribution and maintenance. Questions distinguish observed, reported and inferred data; request descriptions of labels, sampling strategies and representativeness; and ask what cleaning occurred and whether original data and processing software remain available. Suitability and use restrictions are documented separately. Maintenance questions identify owners, correction mechanisms and how additions, deletions and labeling updates reach consumers. Its word-order example shows why a transformation suitable for one task can make data unsuitable for another.

3. Cross-validation and held-out evaluation

   Testing on data used to fit or repeatedly tune a system can overestimate generalization. The documentation separates training, validation for selection, and an untouched final test set; cross-validation does not eliminate the need for final held-out evaluation. Applied to model judges, tune rubrics and examples on calibration data, freeze the judge, then measure agreement with independent human labels on unseen examples. Applied to context policies, select retrieval and compression settings on development tasks and evaluate the selected policy on held-out tasks. Repeatedly changing settings after seeing test results converts that test set into development data.

4. Training Krea 2 - What matters in generative model training.

   Curate against the intended aesthetic coverage rather than relying exclusively on conventional aesthetic or image-quality scores.

5. Krea 2 Technical Report

   Krea's team describes curation for creative exploration, where stylistic diversity matters and blur can be an intentional artistic choice. Its filtering therefore considers image-caption alignment and representability at the training resolution rather than simply retaining the highest aesthetic scores. The team also reports that cluster-based sampling can discard individual named entities within broad visual categories. It checks captions against a Wikipedia-derived concept list, prioritizes rare concepts, and repeats the check after sampling to identify concepts lost during selection.

6. PROV-DM: The PROV Data Model

   Provenance records the entities, activities and responsible agents involved in producing or delivering information. Entities include files and document versions; activities use and generate entities; derivation connects an output to an earlier entity through transformation, update or construction. These relationships provide a concrete representation for source-to-derived-data lineage. PROV distinguishes a particular document version from a resource identifying its changing latest version and warns that provenance descriptions must remain valid as resource state changes.

7. Deduplicating Training Data Makes Language Models Better

   Lee and colleagues compared exact repeated-substring removal with approximate document deduplication. Exact-substring removal eliminated 7.18% of C4 training tokens; near-duplicate matching found training counterparts for 4.60% of C4 validation examples. In their 1.5-billion-parameter models after two epochs, the fraction of generated tokens belonging to copied training sequences fell from 1.571% on original C4 to 0.168% after exact-substring deduplication. The generation test used 100,000 unprompted sequences, each up to 512 tokens, and counted matches of at least 50 tokens.

8. Beyond Accuracy: What Data Quality Means to Data Consumers

   Wang and Strong's Spring 1996 paper investigated data quality through surveys of data consumers and subsequent sorting studies. It organized quality into intrinsic, contextual, representational and accessibility categories. The motivation was that improvement efforts focused too narrowly on accuracy, overlooking whether people could interpret, access and use data for their tasks. This establishes a pre-AI foundation for purpose-dependent curation.

9. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models

   DataComp-LM addresses a comparison problem: changing data, model architecture and compute simultaneously makes improvements difficult to attribute to curation. Its workflow fixes a scale-specific training recipe, varies filtering or data mixtures, then assesses resulting models on downstream tasks. In the reported experiments, classifier-based filtering was effective, but the choice of positive and negative examples substantially affected filter usefulness. This makes dataset selection an empirical engineering decision rather than treating a filter's quality score as an intrinsic property of text.

10. On the Two Different Aspects of the Representative Method

    Neyman's 1934 paper contrasted stratified random sampling with purposive selection intended to reproduce known population characteristics. He showed through mathematical analysis and constructed populations that matching selected control averages need not preserve other quantities or distributions. His argument favored sampling procedures whose uncertainty could be assessed over repeated samples, rather than judging representativeness from a sample's apparent resemblance alone. Time and resource constraints motivated studying samples instead of entire populations.

11. A Theory for Record Linkage

    Ivan Fellegi and Alan Sunter, at Canada's Dominion Bureau of Statistics, formalized linkage between records representing the same person, object, or event. Their December 1969 paper addressed maintaining administrative files and combining information across collections. Recording errors mean one entity can produce different records, while different entities can produce identical recorded attributes. Their decision rule distinguishes links, non-links, and unresolved pairs requiring further review. Under its probability model, it minimizes unresolved decisions at specified false-link and missed-link error levels.

12. On the Resemblance and Containment of Documents

    Broder's work at Digital Systems Research Center addressed near-identical web documents when comparing every full document pair was impractical. It used compact document sketches and reported application to an AltaVista crawl. Beyond the shingling mechanism, the paper identifies two useful limitations: larger shingles increase sensitivity to small edits, and strong resemblance is not transitive. Successive revisions can each resemble their predecessor while distant revisions differ substantially. Pairwise similarity therefore does not make every member of a connected duplicate cluster interchangeable.

13. Datasheets for Datasets: March 2018 Working Draft

    The March 23, 2018 working draft proposed dataset datasheets by analogy with electronic-component documentation, aiming to communicate construction, appropriate uses and limitations between creators and users.

14. Apache Beam glossary

    Event time describes when an event occurred, according to its timestamp; processing time describes when a pipeline processes it. Events need not arrive in event-time order. A user interaction can therefore have an occurrence time different from the time its record is handled.

15. HL7 Terminology: DataAbsentReason

    HL7 distinguishes a value that is unknown, temporarily unknown, not asked, declined, masked for security or privacy reasons, inapplicable, or unsupported by the source system. These codes explain why expected content is absent. They provide a concrete example of preserving missingness reasons instead of collapsing different collection circumstances into one blank value.

16. ODRL Information Model 2.2

    ODRL represents permissions, prohibitions and duties associated with use of an asset. Rules identify actions, involved parties and applicable constraints. Machine-readable policies can be linked to the content they describe so consumers can retrieve its terms of use and reuse.

17. Scaling Enterprise-Grade RAG Systems: Lessons from the Legal Frontier

    Storage segregation and document retention must be configurable for the customer and use case.

18. Statistics Canada Quality Guidelines: Coverage and frames

    The target population comprises the units about which information is wanted. Practical exclusions can narrow this to a survey population. A sampling frame identifies and provides access to units; omissions, duplicates, erroneous inclusions and misclassifications create coverage errors. Probability sampling from a restricted survey population does not automatically justify claims about the broader target population. The guidance recommends documenting these differences and periodically checking frame coverage against other sources.

19. Statistics Canada: Survey Methods and Practices

    Probability sampling uses random selection with calculable, nonzero inclusion probabilities. Stratified sampling partitions a population into mutually exclusive groups and samples within them, supporting subgroup estimates. Judgment sampling deliberately selects cases and can help exploratory work, but its unknown selection probabilities limit population inference. Design weights are inverses of inclusion probabilities. Disproportionate subgroup sampling therefore changes sample proportions and requires the sampling design to be reflected in population estimates.

20. Computer Use at the Edge of the Statistical Precipice

    Vary task data, appearance, and initial state across runs, while checking that every generated combination remains valid.

21. TensorFlow Data Validation: Checking and analyzing your data

    TFDV compares dataset statistics with expectations such as types and permitted categorical values, supports checks before and after transformations, and compares training with serving data or consecutive data spans. Automatically inferred schemas capture basic properties and require user review. Distribution visualizations expose missing values and suspicious patterns, but an imbalanced feature can occur naturally. Drift thresholds require domain knowledge and experimentation.

22. NIST: Choosing a Sampling Plan with a Given OC Curve

    NIST models defect counts in a random sample using a binomial approximation when the collection is much larger than the sample. Substituting zero defects into its formula gives P(no defects observed)=(1-p)^n, where p is the collection's defect fraction and n is the sample size. Constructed illustration: with a 1% defect fraction, inspecting 100 randomly selected records still has about a 37% chance of finding none. A clean audit sample therefore does not establish a defect-free collection.

23. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift

    The paper separates detecting an input-distribution change, identifying examples characteristic of that change, and determining whether prediction quality worsened. Distribution changes can be benign. Conversely, compressing inputs into monitored representations can hide changes because distinct inputs may map to the same representation. A classifier distinguishing reference from current data can surface characteristic records for inspection, but directly assessing prediction error on current data requires labels.

24. Statistics Canada Quality Guidelines: Imputation

    Imputation assigns replacement values to missing, invalid or inconsistent data after applicable follow-up and correction. Statistics Canada recommends reproducible methods and audit trails, retaining original and imputed values, and recording which values changed, the method and any donor records. Treating imputed values as observations can mislead. Aggregate stability can conceal opposing changes in different subgroups, so assessment should examine affected domains and relationships as well as overall quantities.

25. Databricks: Expectation Recommendations and Advanced Patterns

    The documented quarantine pattern marks records violating explicit rules and exposes separate valid and invalid views, retaining failed records for separate processing. Its taxi example tests whether pickup and drop-off ZIP fields are present. The documentation also distinguishes validation from orchestration: a separate validation table does not automatically prevent downstream tables from updating. Enforcing a release gate requires an execution dependency that actually waits for validation success.

26. Unicode Standard Annex #15: Unicode Normalization Forms

    Unicode distinguishes canonical equivalence from weaker compatibility equivalence. NFC preserves distinctions between compatibility-equivalent characters, whereas NFKC and NFKD fold those distinctions. Compatibility normalization can remove formatting or distinctions important to meaning and should not be applied indiscriminately. Normalizing representation is therefore a task-dependent transformation, not automatically a harmless repair.

27. RAG Evaluation Is Broken! Here's Why (And How to Fix It)

    Entity identity and attribute cardinality must be resolved consistently during ingestion and query interpretation.

28. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

    The authors examined documents excluded by C4's word blocklist, finding scientific, medical and other non-offensive content among removals. Identity-mention analysis and manual inspection showed exclusion of benign material discussing sexual orientations. A dialect-model analysis found disproportionate removal of text assigned African American English and Hispanic-aligned English categories. Inspecting rejected records therefore revealed composition effects that examining only the retained corpus would miss.

29. Introduction to Information Retrieval: Near-duplicates and shingling

    A k-shingle is a contiguous sequence of k terms; each document becomes a set of such sequences. Jaccard similarity is |A intersection B|/|A union B|. A threshold identifies substantial textual overlap. MinHash estimates this similarity because, under a random permutation, the probability that two sets share the minimum element equals their Jaccard similarity. Exact fingerprints and approximate overlap serve different purposes. Application inference: keep document identity, source location, revision identifier, content hash, validity and provenance separately from duplicate-cluster membership. A small change to a date, number or negation can be material despite high overlap.

30. CloudEvents Specification v1.0.2

    CloudEvents requires producers to make the combination of source and id unique for each distinct event. A retransmitted duplicate may retain its id, and consumers may treat matching source–id pairs as duplicates. This supplies an explicit event identity rule independent of whether two payloads contain similar text.

31. Training Krea 2 - What matters in generative model training.

    Apply inexpensive hash-based deduplication before more costly embedding-based duplicate detection.

32. TRIPOD+AI expanded checklist: timing, labels, and evaluation separation

    Specify the intended decision point t0, eligible population, predictor definitions and measurement times, and outcome assessment procedure. For prognosis, define the horizon h, such as an event within 28 days after t0. For diagnosis, identify the reference standard used to determine condition status. Predictors should be measured before or when the model is intended to run; assessment and blinding procedures must avoid outcome leakage. Report participant overlap, including repeated records, across training, tuning, and evaluation. Engineering interpretation: patient-disjoint evaluation asks about unseen people; a later-period evaluation asks about performance as time and practice change; evaluation at another institution asks about transport across settings. These dimensions can be combined, and their dates, populations, and differences must be explicit.

33. scikit-learn: Common pitfalls and recommended practices

    Preprocessing learned from data must be fitted using training data alone and then applied to held-out data. Even an average used for normalization leaks information if computed over the test subset. The documentation applies this warning to feature selection, scaling, imputation and PCA, and demonstrates misleading above-chance performance on random labels when feature selection sees test data. Pipelines help fit transformations on the correct subset during cross-validation.

34. SWE-rebench: Lessons from Evaluating Coding Agents on Real Software Engineering Tasks — Ibragim Badertdinov, Nebius

    An agent can recover the solution through retained future Git history or public repository access, and disabling one retrieval tool does not close equivalent paths.

35. Analyzing Dataset Annotation Quality Management in the Wild

    The paper recommends iterative annotation batches: pilot the scheme and instructions, inspect quality, then correct labels, train annotators or revise guidelines. A scheme defines available labels; guidelines explain when to apply them. Multiple judgments support agreement checks and adjudication, meaning resolution into a final annotation. Expert inspection and error-rate audits remain distinct from agreement measurement. Major guideline changes can require reannotation of earlier batches. Retaining individual annotations supports reproducibility and preserves information lost during aggregation.

36. Google: Supervised learning foundations

    Features are input values x; a label y is the target recorded for an example. Parameters θ are learned numbers defining a model fθ. Training repeatedly compares predictions with labels using a loss and adjusts these numbers to improve agreement. Inference evaluates the trained function on a new input without needing its label. Generalization means performing well on examples outside training; evaluation therefore compares predictions against withheld labels. Programmer-oriented illustration: training fits w and b in f(x)=wx+b from examples, whereas an authored rule such as if x>c executes a programmer-specified condition. Both execute code at inference, but the origin of their decision behavior differs.

37. Stripe card-dispute timing and lifecycle

    A dispute becomes observable to a merchant when Stripe notifies it through configured channels or the API, after the account owner contests a payment. Cardholders usually have 120 days to initiate a dispute, with exceptions; future-service windows may start at the service date. Early fraud warnings and inquiries can precede formal disputes, but warnings can also arrive afterward. A case can await evidence or issuer review before its status becomes won or lost; rare late wins can revise a loss. An inquiry closed without escalation is not an explicit issuer win. Label-design implication: preserve payment time, observation time, reason and evolving status. At a chosen cutoff, an unresolved case remains unresolved, and no recorded dispute means only that none has been observed by then, not confirmed non-fraud.

38. Why Your Agent Disagrees With Itself (And What To Do About It)

    The password-spray example distinguishes failed attempts from successful access and combines that evidence with the customer's notification policy.

39. Right censoring and unresolved event outcomes

    Let T be event time and C the end of observation. A right-censored record stores U=min(T,C) and an indicator delta showing whether the event was observed. When delta=0, the record establishes only that the subject remained event-free through C; the later outcome is unknown. Survival models use this partial information rather than treating C as an event time. S(t)=P(T>t) describes survival beyond t. Financial application: for a default-within-H label, an account observed without default only until C<H is unresolved, not a negative example. A negative requires sufficient event-free follow-up through H.

40. Zendesk: What is the difference between a solved ticket and a closed ticket?

    Agents can mark a ticket solved after resolving the issue or completing work. Closure is performed through automation or triggers. A reply can reopen a solved ticket, but a reply to a closed ticket creates a follow-up while the original remains closed. Thus, for this documented workflow, a closed status records a workflow transition and does not by itself establish customer-confirmed resolution.

41. Obermeyer et al.: Healthcare cost as a biased proxy for need

    The studied algorithm ranked patients for extra care using predicted healthcare spending as a proxy for health need. Black patients were sicker than White patients at the same score. The authors linked this mismatch to lower spending on Black patients at comparable illness burden, reflecting unequal access and utilization. Predicting the chosen cost label could therefore work while allocation by that score underserved patients with greater need. Their estimated correction increased the Black share selected for additional help from 17.7% to 46.5%. The mechanism is target mismatch: accurate prediction of resource consumption does not establish accurate measurement of need.

42. Learning Classifiers from Only Positive and Unlabeled Data

    Charles Elkan and Keith Noto distinguish an example's actual class from whether someone has labeled it. Their biological-database example contains known qualifying proteins but no explicit collection of nonqualifying proteins. Records absent from that specialized database can include both undiscovered positives and negatives. Consequently, absence from the curated collection cannot simply become a negative label. Their proposed learning methods explicitly model the relationship between class membership and label availability.

43. Inter-Coder Agreement for Computational Linguistics

    Inter-annotator agreement measures consistency of judgments on the same items, not their independent correctness: annotators can share a mistaken interpretation. The paper's mathematical examples show why category prevalence matters. If two independent random labelers each assign one category 95% of the time, their expected agreement is 90.5%, despite having no item-specific judgment ability. Chance-corrected measures use different assumptions about category frequencies and annotator behavior, so their values require interpretation alongside the labeling task and distributions.

44. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

    For subjective annotation tasks, majority aggregation can hide systematic differences between annotators and discard internally consistent perspectives. The paper preserves annotator-specific labels and evaluates methods across seven subjective tasks. Its findings support investigating disagreement rather than automatically treating minority judgments as random errors. The authors recommend retaining individual annotations where feasible, with appropriate consent and anonymization.

45. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

    Northcutt, Athalye, and Mueller used automated error detection followed by human review rather than automatically replacing suspicious labels. For ImageNet's 50,000-example validation set, their method flagged 5,440 candidates. Five workers reviewed each candidate: 2,524 retained support for the original label, and 1,428 received a majority-supported replacement. Other cases involved multiple plausible labels, neither offered label, or unresolved disagreement. Their inspection also found valid unusual viewpoints and cropped objects among automated false alarms.

46. Snorkel: Rapid Training Data Creation with Weak Supervision

    Weak supervision uses cheaper, imperfect label sources such as rules, external knowledge bases and weak classifiers. Snorkel's 2017 system lets domain experts express these sources as labeling functions that return a label or abstain. It combines their overlapping and conflicting outputs into probabilistic labels. Modeling correlations matters because similar heuristics can otherwise be counted as independent support. Although its label-combination step uses unlabeled training examples, the paper explicitly assumes a small labeled development set and a blind, held-out labeled test set.

47. Quality and Efficiency of Manual Annotation: Pre-annotation Bias

    Marie Mikulová and colleagues compared manual correction of parser-generated annotations with annotation from scratch for Czech dependency syntax: identifying which word each word depends on and the relationship type. Four annotators worked across eight setups without annotating the same material twice. Annotation from scratch took almost 1.7 times as long on average. Accuracy against adjudicated reference annotations remained broadly similar, while pre-annotation increased agreement. This demonstrates a setting where automated suggestions reduced work without the observed quality loss that concern about suggestion bias might predict.

48. Bias in the Loop: How Humans Evaluate AI-Generated Suggestions

    Jacob Beck and colleagues randomized annotation-workflow conditions for 2,784 participants checking emissions values against corporate-report tables. Suggestions were manually manipulated and presented as AI outputs. Requiring reviewers to enter a replacement value when rejecting a suggestion reduced corrections: reviewers accepted more incorrect suggestions but also rejected fewer correct ones. Aggregate accuracy was slightly higher despite increased acceptance of errors. The experiment therefore exposes a consequential distinction between overall accuracy and the particular mistakes a curation workflow permits.

49. When Machines Mislead: Human Review of Erroneous AI Cheating Signals

    Duolingo researchers William Belzak, Chenhao Niu, and Angel Ortmann Lee inserted fabricated copy-typing alerts into previously certified exam sessions. Each of two studies sampled 170 sessions. Model-estimated rejection of fabricated alerts increased from 50% under the original guidelines to 71% after guidelines emphasized corroborating video evidence. Thus human review rejected many false suggestions but also accepted some. Operational alerts were rejected more often after the revision, although their underlying correctness was not independently established.

50. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    The When Machines Mislead case study inserted fake copy-typing alerts into legitimate historical exam sessions and found that skilled proctors accepted half of those alerts.

51. A Sequential Algorithm for Training Text Classifiers

    David Lewis and William Gale at AT&T Bell Laboratories studied uncertainty sampling to reduce the labeling needed for text classification. Their loop trains a classifier, selects unlabeled examples whose categories are uncertain, obtains their labels, and retrains. In their 1994 newswire experiments, uncertainty selection generally reached useful classification performance with fewer labeled examples than random selection or selecting the most likely positives. This separates choosing informative examples for learning from choosing examples representative of population proportions.

52. Hierarchical Sampling for Active Learning

    Sanjoy Dasgupta and Daniel Hsu's 2008 paper shows why uncertainty-driven labeling can overlook confidently misclassified regions. In its one-dimensional teaching example, an initial sample misses a small group; subsequent queries remain near the current decision boundary, so the overlooked group never corrects the learner's mistaken confidence. Their alternative uses hierarchical clusters and random sampling within selected regions, updating label statistics and refining the partition. It combines informative selection with explicit management of sampling bias.

53. Why Your Agent Disagrees With Itself (And What To Do About It)

    Active learning selects potentially problematic examples for human clarification, then incorporates corrected labels or additional features into the next training cycle.

54. Apache Iceberg Table Specification

    An Iceberg snapshot represents a table state at a particular time. Immutable manifest files list data or delete files with partition and tracking metadata; a manifest list identifies the manifests belonging to a snapshot. Membership is established through explicit metadata rather than simply reading everything currently present in a directory. Readers retain their selected snapshot until refreshing metadata.

55. DVC: dvc.yaml Files

    DVC separates pipeline declarations in dvc.yaml from recorded pipeline state in dvc.lock. Lock entries include stage commands, dependencies, outputs, content hashes and parameter values. These records support detecting changed stages and identifying the data versions associated with pipeline outputs. A recipe and its recorded input/output state therefore provide complementary parts of reproducibility.

56. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    Lineage must survive graph mutation: entity merges retain both source sets, and invalidation records the new evidence responsible for the change.

57. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Penedo and colleagues trained an educational-content filter using model-generated ratings focused on grade-school and middle-school knowledge. They compared filtering thresholds using 1.71-billion-parameter models trained on 28 billion tokens; threshold 3 provided the best average performance in those experiments. Filtering also changed topic composition: FineWeb-Edu increased representation of education and history while reducing business and entertainment. The filter's reported validation F1 measured agreement with Llama-generated annotations, whereas downstream model experiments assessed a different question: whether the selected text improved the tested capabilities.

58. scikit-learn confusion_matrix

    A confusion matrix counts reference classes in rows and predicted classes in columns. With acceptable=1 and unacceptable=0, false acceptance is C[0,1] and false rejection is C[1,0]. Conditional rates are C[0,1]/(C[0,0]+C[0,1]) and C[1,0]/(C[1,0]+C[1,1]); a zero denominator leaves that rate undefined. Applied to judge validation, freeze the rubric and acceptance threshold, obtain independent reference labels, and compare judge decisions on held-out examples. Keep unresolved human disagreements and abstentions separately visible.

59. FineWeb-Edu dataset card

    The dataset card reports that educational-score threshold 3 retained approximately 1.3 trillion tokens, while threshold 2 retained 5.4 trillion. These are retained corpus volumes, not quantities consumed in one training experiment. Its changelog distinguishes additions from repairs: version 1.3.0, dated January 31, 2025, corrected documents skipped during processing and added approximately 35 billion tokens across listed snapshots; other releases added newly collected snapshots. Earlier versions remain available through named branches.

60. Databricks: Use Change Data Feed

    Change data feed exposes row-level inserts, deletes and updates between table versions. Update records distinguish values before and after modification, with commit versions and timestamps. Consumers can use these records to update downstream datasets incrementally. The feed is not a permanent history: changes become unavailable when their retained versions expire, and restarting from an unavailable version fails. A reproducible maintenance process therefore needs to account for both what changed and whether the required history remains accessible.

61. BigCode: The Stack Datasets and Removal Process

    The Stack's release history records changes to included licenses and removal requests: version 1.1 excluded requests submitted by November 15, 2022, and version 1.2 excluded requests submitted by February 9, 2023. Its process retains validated removal requests and code pointers to prevent reinclusion. Downloaders must agree to use the most recent allowed version. The page also states that original code-license terms continue to apply and that record provenance supports attribution. Thus an older reproducible release is not automatically an allowable rollback target.

62. Machine Unlearning

    Removing records from a stored dataset is distinct from removing their influence on an already trained model. The paper explains that successive training updates depend on earlier updates, so a record's contribution is not generally undone by deleting its original row. It formulates removal from learned models as a separate technical problem. For curators, this supports identifying affected model artifacts and handing off remediation rather than declaring downstream effects reversed after source deletion.

63. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    The human-AI interaction loop is cyclical: interfaces that encourage rubber-stamping can turn model-influenced approvals into misleading evaluation and training labels.

64. Build Your First Demand-Driven Context Base: Let AI Agents Tell You What They Need

    They provide maintenance signals, but the Q&A leaves semantic correctness and economical refresh at scale unresolved.

65. Your Evals Are Meaningless (And Here’s How to Fix Them)

    Dataset drift, as used in the talk, includes inadequate coverage of messy, ambiguous, and combined real-user requests.

66. Task Fidelity Scaling Laws — Kobie Crawford, Snorkel AI

    Snorkel describes shared rubrics and expert-provided ground truth as the basis for checking agreement among humans and between humans and LLM judges.

67. Why Your Agent Disagrees With Itself (And What To Do About It)

    The speaker argues that unstable examples tend to occupy the decision boundary, or gray zone, where labels or available information are ambiguous.

68. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

    Include the reason for each verdict, especially when the judge must learn a complex policy.

69. Build Your Own Deep Research Agent + Technical Writer

    The demonstrated judge passed every simulated online trace despite labeled failures, showing that its small-split results did not establish reliable behavior on those scenarios.

70. Weak supervision needs evaluation beyond supervisor agreement

    Oversight becomes difficult when an evaluator cannot reliably judge behavior available to a stronger model: matching its labels can reproduce its mistakes. The paper trains a weak model on ground-truth labels, obtains its predictions on held-out examples, and trains a stronger student on those weak labels. It compares ground-truth test performance with both the weak supervisor and a strong model trained directly on ground truth. Performance gap recovered is PGR=(student-weak)/(strong_ceiling-weak), requiring a nonzero denominator. PGR=0 means no improvement over the supervisor; PGR=1 matches the experimental ceiling. Independent correctness scores and agreement on supervisor-wrong examples distinguish useful generalization from imitation. Experiments show task-dependent generalization and overfitting to weak labels; an auxiliary confidence loss improves some NLP results.

71. NIST AI RMF: Monitoring, incident response, and recovery

    NIST calls for ongoing risk tracking, user feedback, documented incident response and recovery, change management, and assigned authority to override or deactivate systems inconsistent with intended use. It explicitly considers tracking risks when suitable metrics are not yet available. Engineering application: define escalation owners and suspension criteria, preserve incident and version records, restore an evaluated safe operating mode, and evaluate changed behavior before resuming. When clinical labels arrive late, monitor input availability, operational failures, and reported incidents immediately, then calculate outcome metrics on cohorts whose follow-up has matured. Pending labels must remain unknown rather than being counted as negative outcomes or successful predictions.

72. Training Krea 2 - What matters in generative model training.

    Match sample complexity to training resolution, then use curated mid-training and supervised fine-tuning data to shape the model toward downstream uses.

73. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    A synthesized fact can hide both its original wording and the authority of its actual source, so retain verbatim inputs and explicit links to derived artifacts.

74. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    Project ingestion metadata onto derived artifacts so retrieval can filter by source classification.

75. Computer Use at the Edge of the Statistical Precipice

    Confidence intervals should account for both action stochasticity and environment variation while respecting the benchmark's structure.

76. Data and Environment Curation for Post-training LLMs

    Treat curation as a sequence of experimentally tested choices, including source mixing, question filtering, teacher selection, answer filtering, and answer multiplicity.

77. Data and Environment Curation for Post-training LLMs

    Sampling multiple answers per question reportedly helped both reasoning-model and agent-data curation.

78. Data and Environment Curation for Post-training LLMs

    Answer filtering reportedly worked poorly in the reasoning-data work, while synthetic rewriting and task augmentation worked poorly in the agent-data work.