Angel Ortmann Lee is a software engineer and researcher focused on automation bias in high-stakes AI. At Duolingo, she worked on security for the Duolingo English Test, investigating how automated cheating detection can distort human judgment and unfairly influence decisions affecting test takers.
She studied at the University of Southern California, where she was named to the engineering school’s dean’s list. In 2025, she coauthored research on keystroke and mouse behavior for detecting copy-typing and potential cheating rings, alongside a study of erroneous AI-generated cheating alerts. Together, the papers examine both the detection of suspicious behavior and the dangers of treating algorithmic suspicion as proof.
- Human oversight must withstand automation bias. When fabricated copy-typing alerts were introduced into legitimate historical test sessions, experienced reviewers accepted approximately half. Requiring independent corroborating evidence increased rejection of false alerts from roughly 50% to 71%, without changing the underlying model. The experiment did not affect actual test takers.
- Separate detection from policy judgment. A system might correctly identify an object resembling headphones without establishing a testing violation: a hearing aid, for example, should not be treated as misconduct. Distinguishing those decisions protects test takers and produces more accurate feedback about model performance.
- Human disagreement improves training data. Reviewers who reflexively approve predictions can turn model errors into misleading labels. Capturing corrections, rejected recommendations, and the reasoning behind disagreements gives developers stronger evidence for evaluation and subsequent improvement.
- Match review friction to the stakes. In her AI Engineer World’s Fair presentation, Ortmann Lee extended these principles to writing tutors and coding agents: feedback should attach to specific text, while coding assistants should surface assumptions, propose reviewable plans, and break work into manageable changes. High-stakes decisions need deliberate checkpoints; low-risk interactions can remain streamlined.
Her practical focus is designing for discernment: structuring AI-assisted decisions so people investigate evidence, retain responsibility, and generate feedback that improves the system.
