Design Engineering and AI Interfaces: Giving People Control Over AI That Acts

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. NIST Handbook 161: Usability Handbook for Public Safety Communications

   The handbook separates interaction design—what the system communicates and when—from interface design—how it communicates. Its fingerprint-device example distinguishes capture, submission, returned results, and errors, asking what feedback lets the operator understand each stage and continue after a problem. It recommends documenting these exchanges in an interaction specification and iterating designs with users before implementation becomes difficult to change.

2. Cognitive Engineering

   Norman distinguishes translating a person's goal into available actions from interpreting the resulting system state against that goal. The gulf of execution concerns the first difficulty; the gulf of evaluation concerns the second. Designers can reduce both by matching controls and displays to users' intentions. His faucet example separates the desired variables—temperature and total flow—from controls that independently regulate hot and cold water. Delayed feedback can make evaluation harder because the person may no longer remember the relevant action.

3. Your coding agent doesn't always follow your rules

   The speaker recommends treating instructions and verification as separate responsibilities, even with strong specifications, MCP servers, sub-agents, and context.

4. Building Multi-agent Systems with Finite State Machines

   Place an explicit approval state between the model's proposed tool call and the application's next action.

5. Guidelines for Human-AI Interaction

   The guidelines recommend communicating capabilities and likely mistakes, showing task-relevant information, and making assistance easy to invoke, dismiss, and correct. Under ambiguous goals, the system should clarify or reduce its scope. Correction includes editing, refining, and recovering rather than requiring another complete attempt. Other recommendations preserve recent interaction context, limit disruptive adaptation, support granular feedback and global controls, and explain how user actions affect future behavior. Table 1 illustrates correction with an editable reminder and uncertainty handling with selectable autocomplete alternatives.

6. Principles of Mixed-Initiative User Interfaces

   Horvitz combines automation with direct manipulation: users can invoke, terminate, and refine automated assistance. The proposed principles account for uncertainty about intent, interruption costs, and the consequences of incorrect action; clarification should resolve important uncertainty without needlessly bothering users. LookOut provides a concrete example: it extracts appointment details from email, displays editable guesses, and lets the person save the result. When it cannot identify an exact appointment, it can show a relevant calendar period for manual completion.

7. Signifiers, not affordances

   An affordance concerns actions possible for a particular actor in an environment; it need not be visible or known. A signifier is a perceivable cue that communicates meaningful information, including possible actions or current conditions. Norman distinguishes the available action from the evidence that helps someone discover or interpret it. Signifiers can be deliberately designed or incidental, and their interpretation can be unreliable.

8. Direct Manipulation: A Step Beyond Programming Languages

   Shneiderman's August 1983 paper describes direct manipulation through continuously visible objects, actions or labeled controls instead of complex command syntax, and rapid, incremental, reversible operations with visible effects. This lets users inspect whether an action advances their goal and change direction. The paper also warns that graphics can confuse: icons require learning, representations can imply unavailable operations, and visual layouts can consume excessive space.

9. Figma: Rewrite, translate, and shorten text with AI

   Figma's documented writing tools begin with selecting a text layer. Rewriting adds a prompt to describe the desired change; shortening uses a named action; translation uses a language selection. These provide concrete examples of supplying the target through object selection and the operation through contextual controls, without requiring a separate conversation to identify both. The documented uses include fitting interface copy into limited space and previewing another language.

10. AI and Human Whiteboarding Partnership

    Have the LLM generate a constrained domain-specific language, then deterministically translate that representation into native product objects.

11. AI and Human Whiteboarding Partnership

    AI output is more useful when it becomes ordinary editable product state rather than a flattened image or terminal answer.

12. The era of unbounded products: Designing for Multimodal I/O

    Keep evolving task UI outside the scrolling conversation when users need to return to it or act on it.

13. Collaborate with Claude on Projects

    Anthropic's June 25, 2024 announcement describes Artifacts as a dedicated workspace beside the conversation for generated documents, code, graphics, diagrams, and website designs. Larger code views and frontend previews let people inspect the work product alongside their conversation. This supplies a concrete example of combining conversational direction with a separate artifact surface.

14. When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming

    Mozannar, Bansal, Fourney and Horvitz model an inline suggestion's value as the writing time it may save minus verification, editing and latency costs. Their retrospective analysis uses telemetry from 535 programmers to predict acceptance and selectively withhold likely rejected suggestions. The paper emphasizes that observable acceptance is incomplete: programmers may accept merely to reveal later code, and similar telemetry can arise from different unobserved activities such as thinking, debugging or waiting. It therefore supplies a concrete inline-assistance pattern while warning that acceptance events alone do not establish suggestion quality or user benefit.

15. Building AI Agents that actually automate Knowledge Work

    The speaker associates background automation with more constrained control flow and an explicit review boundary before downstream actions.

16. Magentic-UI, an experimental human-centered web agent

    The prototype combines a visible browser with editable plans and conversation. Before execution, users can revise a plan directly or through textual feedback. During execution, they can pause, take control of the browser, demonstrate a correction, and return control. Action guards request approval for actions deemed consequential, with an option to request approval for every action. Saved plans have a separate gallery where users can inspect and modify them.

17. The Importance of Percent-Done Progress Indicators for Computer-Human Interfaces

    Brad Myers's April 1985 paper distinguishes percent-done indicators from activity displays. A percent-done indicator reports completed work relative to an estimated whole; changing activity without a known total can only indicate that processing continues. In the reported database-query experiment, participants significantly preferred versions with progress indicators, but hypotheses about preferences for constant versus variable response times were not supported. The paper also notes that non-linear or initially unknown work makes percentages difficult to calculate and proposes pass-level, task-level, heuristic, or indeterminate displays instead.

18. Power to the People: The Role of Humans in Interactive Machine Learning

    Amershi, Cakmak, Knox and Kulesza describe interactive machine learning as rapid, focused and incremental cycles in which people provide input, inspect updated behavior and adapt subsequent input. Their reviewed cases show why the complete interaction must be studied: users may supply richer guidance than a label interface permits, repeated system questions can frustrate users and weaken their understanding, and immediate localized output can help users direct corrections at visible errors. They argue that new interaction techniques need evaluation with intended users because the person and learning system influence one another.

19. AIP-151: Long-running operations

    The API pattern returns an operation that clients can use to check progress and retrieve the eventual result instead of treating the initial response as the result. Metadata can describe progress and partial failures. Errors preventing an operation from starting are distinguished from failures during execution. A resource being created or deleted can appear in listings while its state indicates that it is not yet usable. Concurrent operations may be queued, rejected, or superseded under explicitly defined rules.

20. Vercel AI SDK: useChat reference source

    The useChat reference distinguishes submitted, streaming, ready, and error states; ready means idle. Its finish callback includes separate indicators for client abort, disconnection, and streaming error, plus an optional model finish reason. Messages have identifiers, and regeneration can target a specific message. Local message-state updates can occur without an API call. These distinctions support separating displayed content, transport lifecycle, and confirmed task outcomes.

21. HTML Standard: The progress element

    The HTML progress element represents task completion. Determinate progress expresses completed work relative to a total; indeterminate progress indicates ongoing work when the amount remaining is unknown. The value attribute supplies completed work and max supplies the total. Omitting value represents indeterminate progress. The standard's example updates the element from application-supplied progress, making the display dependent on information provided by the application.

22. My AI Thinks I'm Eating My Feelings (and Other Nutritional Insights)

    Split a broad LLM workflow into constrained steps and expose useful partial results before all processing finishes.

23. Progressive Disclosure

    Progressive disclosure initially presents important, frequently needed options and makes specialized options available on request. The split must preserve useful primary functionality, and the route to additional detail needs an obvious control with an informative label. Nielsen distinguishes this from staged disclosure, where people proceed through successive parts of a task that may be equally important.

24. Understanding Success Criterion 4.1.3: Status Messages

    Status announcements need enough context to be meaningful: announcing a changed number alone can obscure what changed. W3C illustrates intermittent progress announcements and complete contextual messages. Removing a visible busy message can communicate availability to sighted users while leaving screen-reader users uninformed; an explicit available-state announcement can supply the missing information. A dialog that receives focus is a context change rather than a status message under this criterion.

25. People + AI Guidebook: Explainability + Trust

    The guide distinguishes explanations of general system behavior from explanations attached to a particular output. It recommends concentrating on information relevant to understanding and decisions rather than attempting to explain everything. Confidence displays require early testing because people differ in their understanding of probabilities. Error responses should provide a way forward, including addressing the current problem and, for high-risk outcomes, moving to manual control.

26. Citations and Trust in LLM Generated Answers

    Participants asked their own questions and received answers with zero, one, or five citations. Displayed citations either came from search results or were randomly selected from previous participants' questions. Citations increased self-reported trust even when randomly selected, although random citations received lower trust than search-derived citations. Checking citations was associated with lower trust. The experiment distinguishes a citation's persuasive appearance from whether it supplies relevant support.

27. Designing for Appropriate Reliance: The Roles of AI Uncertainty Presentation, Initial User Decision, and User Demographics in AI-Assisted Decision-Making

    Cao, Liu and Huang compared no uncertainty display, raw probability, calibrated probability and calibrated-frequency presentations in a skin-lesion classification experiment. Frequency wording described expected outcomes among 100 similar samples. The calibrated-frequency condition changed how participants adjusted confidence in some agreement and disagreement cases, but none of the uncertainty presentations prevented switching to incorrect AI advice; over-reliance occurred in every condition. Participants also did not report better understanding from the frequency display. Thus calibrated uncertainty and its presentation are separate requirements, and neither alone guarantees an appropriate decision.

28. Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions

    Vasconcelos and colleagues compared plain code completions, highlighting tokens with low generation probability, and highlighting tokens predicted likely to be edited. In a mixed-methods study with 30 programmers, edit-likelihood highlighting produced faster completion and more targeted edits and was preferred by participants; low-generation-probability highlighting did not improve the measured outcomes over no highlighting. The study demonstrates that an uncertainty display should identify the user decision it supports: model probability can be a poor proxy for where review effort is useful.

29. “I'm Not Sure, But…”: Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust

    In a September 2023 experiment, 404 retained participants answered eight medical information questions; supplied AI answers were correct on half. First-person uncertainty wording reduced agreement with the system and increased model-estimated participant accuracy from 63.9% to 72.8% relative to responses without uncertainty wording. It also reduced desire to use the system. General uncertainty wording did not produce the same statistically significant accuracy improvement. Between the AI conditions, source-use differences were not significant.

30. Cloud Speech-to-Text: operations.cancel

    Cancellation starts asynchronously and is best effort. Clients must inspect the operation to determine whether cancellation succeeded or the operation completed despite the request. Successful cancellation leaves an operation record carrying a cancelled error status. An unsupported cancellation method can return UNIMPLEMENTED. The empty response to the cancellation request therefore does not establish terminated execution.

31. [Evals Workshop] Mastering AI Evaluation: From Playground to Production

    Expose the judge's rationale alongside its numeric score so reviewers can inspect why it chose the grade and tune the evaluator when its reasoning or calibration is poor.

32. Progressive Disclosure: Designing for Effective Transparency

    Springer and Whittaker's 2018 paper reports two studies comparing global and incremental transparency. Participants initially expected the more incrementally transparent system to perform better, then revised that judgment after use. Qualitative findings indicated that continuous incremental feedback could distract users and interfere with simple working heuristics. The authors use these results to motivate progressive disclosure: begin with simplified feedback needed to operate the system and reveal additional detail as users need it, rather than presenting every internal update at once.

33. How Kepler Built Verifiable AI for Financial Services

    Source attribution alone does not establish source quality or extraction correctness.

34. How Kepler Built Verifiable AI for Financial Services

    Derived numbers need replayable derivation histories spanning source values, calculations, and internal information, not just links to filings.

35. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

    Bansal and colleagues evaluated human-AI decisions across three datasets with 1,626 participants and AI performance chosen to leave room for human-machine complementarity. Every human-AI condition exceeded the compared solo performance, but none of the explanation conditions significantly exceeded the simpler confidence-only condition. Explanations tended to help when the AI recommendation was correct and hurt when it was wrong. The paper distinguishes perceived explanation usefulness and proxy tasks from end-to-end decision accuracy, showing that an explanation can influence agreement without independently supporting the recommendation.

36. Agent Client Protocol: Prompt Turn

    ACP permits a client to request cancellation while a prompt turn is running. The client must answer pending permission requests with a cancelled outcome. The agent should stop model requests and tool invocations promptly, then return a cancelled stop reason after ongoing operations have been aborted and pending updates sent. Updates may still arrive between the cancellation request and that response, and clients should accept them. The specification also recommends immediately marking unfinished tool calls cancelled on the client.

37. UX Design Principles for (Semi) Autonomous Multi-Agent Systems

    Interruptibility should include pausing, checkpointing, rollback, and resumption so users can intervene before mistakes or unwanted resource use compound.

38. React: Synchronizing with Effects

    The documentation's fetching example uses cleanup to ignore results from an obsolete request when the selected input changes. A response for an earlier selection can arrive after a later response; ignoring it prevents stale data from replacing the current display. The documentation distinguishes aborting a fetch from ignoring its result.

39. OpenRefine User Manual: Running OpenRefine

    OpenRefine saves project change history with the data, allowing users to revisit and undo earlier changes after restarting. Selecting an earlier history entry reverses all subsequent changes; making a new edit from that point erases the later redo history. View state has a separate contract: closing a project loses its current filters and facets, while a permalink preserves that view information in the URL. Data recovery and returning to the same working view are therefore distinct capabilities in this implementation.

40. IT Admin for the AI Workforce — Sarthak Aggarwal, Decawork

    Represent the agent actor, accountable owner, represented subject, and delegation context separately.

41. Defying Gravity

    Centralize permission requests and open questions in an inbox, notify the user only when attention is required, and expose progress through task artifacts.

42. IT Admin for the AI Workforce — Sarthak Aggarwal, Decawork

    Agents with authority and side effects need an operational lifecycle beyond model behavior testing.

43. Don't just slap on a chatbot: building AI that works before you ask

    Proactive AI should supplement agency, keep recommendations optional, and make its changes easy to reverse.

44. OWASP Transaction Authorization Cheat Sheet

    Users should identify and acknowledge significant transaction details before authorizing execution. The authorization sequence must not be skipped or reordered, and execution needs a final authorization check. Changing transaction data during authorization should invalidate previous authorization or restart the process. Applied to AI proposals, approval should authorize the inspected scope and values rather than an independently changing proposal.

45. Building Your Own Secure AI Workflows: Human-in-the-Loop Automation with n8n

    Place approval in the execution path so the agent cannot bypass it by choosing to call the underlying tool directly.

46. Building Your Own Secure AI Workflows: Human-in-the-Loop Automation with n8n

    Show the proposed action's concrete parameters in readable form, rather than only its tool name.

47. Harder to Ignore? Revisiting Pop-Up Fatigue and Approaches to Prevent It

    Researchers varied repeated exposure to dialogs before changing a field to contain an actionable instruction. Repetition reduced responses to that change in the ordinary dialog. Requiring interaction with the relevant field through swiping or typing resisted habituation better than some attention-grabbing animations and delays. Typing imposed the greatest interaction burden. The study supports testing whether people still inspect consequential details after repeated confirmations, rather than treating the presence of a dialog as evidence of attention.

48. When Machines Mislead: Human Review of Erroneous AI Cheating Signals

    The researchers introduced fabricated copy-typing alerts into previously certified test sessions and examined proctor decisions in two studies. Revised instructions explicitly required independent suspicious behavior before upholding an alert. Model-estimated rejection of fabricated alerts increased from 50% to 71%; rejection of genuine operational alerts also increased. The study demonstrates a way to investigate whether reviewers challenge erroneous assistance instead of measuring approval alone.

49. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    The When Machines Mislead case study inserted fake copy-typing alerts into legitimate historical exam sessions and found that skilled proctors accepted half of those alerts.

50. Compensating Transaction pattern

    Compensation performs application-specific actions to address effects of already completed work. Restoring an old snapshot can overwrite concurrent changes, and compensation need not restore the original state. Compensation can itself fail and require resumable progress or manual intervention. The travel example distinguishes offering another hotel from cancelling successfully booked flights; cancellation may not provide a full refund. Irreversible steps need explicit recognition.

51. From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik

    The compensation pattern, also called the Saga pattern, pairs execution with explicit undo operations and invokes them in reverse completion order.

52. RFC 9110: HTTP Semantics

    HTTP conditional requests can prevent one client from overwriting another client's concurrent work. If-Match compares supplied entity tags with the target representation before applying a change. When the condition fails, the server must not perform the requested method and may return 412 Precondition Failed. The standard separately permits reporting success when the requested change already appears to have occurred. Version checking therefore provides a concrete implementation boundary between editing an inspected version and silently replacing newer work.

53. What Do Prototypes Prototype?

    Houde and Hill organize prototyping around questions about an artifact's role, look and feel, and implementation. Role concerns its usefulness in a person's life; look and feel concerns the experience of interacting; implementation concerns how it works. The unresolved question should determine the prototype. Visual polish and the tools used do not establish completeness or technical feasibility, and teams must explain which aspects represent the eventual system and which do not.

54. Storybook: Mocking network requests

    Storybook documents using Mock Service Worker to intercept network requests and return controlled responses. Its document-screen example exposes loading, success, and error states. Separate stories provide successful data and a delayed HTTP 403 response. This is a published fixture for making otherwise brief or difficult-to-reproduce interface states available for inspection.

55. Wizard of Oz Experimentation for Language Technology Applications: Challenges and Tools

    A Wizard-of-Oz prototype has a human simulate some or all system functions, allowing researchers to explore interactions before engineering the complete capability. The paper develops and evaluates WebWOZ tooling and identifies consistency of wizard performance as a methodological concern. Such prototypes can investigate a proposed user experience while leaving actual automated capability unimplemented.

56. Full Walkthrough: Workflow for AI Coding — Matt Pocock

    Check whether the first proposed slice produces observable behavior; an instruction to use vertical slices does not guarantee that the generated tasks actually do so.

57. State Chart XML: State Machine Notation for Control Abstraction

    A basic state machine identifies its active state and the transitions that state permits in response to events. A matching transition selects a target state; conditions can further restrict when it is taken. Events may originate inside or outside the system. SCXML also distinguishes actions performed when leaving a state, during a transition, and when entering the next state. This supplies precise vocabulary for connecting visible states, user actions, and execution acknowledgments.

58. Building Multi-agent Systems with Finite State Machines

    Use actors for communication and encapsulated execution, statecharts for behavioral structure, and LLMs for context-dependent reasoning.

59. Your Agent Didn’t Fail. Your Harness Did.

    Fluent output does not establish that the harness assembled a complete or current working set.

60. GOV.UK Service Manual: Using moderated usability testing

    Moderated usability testing observes actual or likely users attempting specific tasks. Think-aloud requests help reveal what participants understand and expect. Tasks should present believable goals without revealing the intended solution. Teams should agree research questions and target users, observe with neutral instructions, and ask follow-up questions about unclear behavior. Assistive-technology users often benefit from using their own configured devices. Dummy data can be useful but may reveal fewer contextual issues than appropriately protected real data.

61. Production Evals For Agentic AI Systems

    Apply an SRE or production-engineering lens: assess delivered value, operational reliability, human burden, risk, user experience, scalability, and resilience.

62. Psychological Factors Influencing Appropriate Reliance on AI-enabled Clinical Decision Support Systems: Experimental Web-Based Study Among Dermatologists

    In this web experiment, 223 dermatologists classified 24 cases before and after seeing AI advice; five recommendations were deliberately incorrect. The analysis distinguished accepting useful advice, rejecting incorrect advice, and changing a previously correct answer into an incorrect answer. Some initially correct judgments became wrong after erroneous advice, while correct advice was also frequently rejected. This provides a concrete way to test oversight behavior rather than counting approvals alone.

63. The Theory Behind Heuristic Evaluations

    Heuristic evaluation has evaluators inspect an interface against recognized usability principles and explain the problems they identify. Evaluators inspect independently before combining findings because different people discover different problems. Nielsen distinguishes this from user testing: in a user test, researchers interpret participants' attempts to perform tasks; in heuristic inspection, the evaluator directly analyzes the design. Inspectors may receive domain explanations that would obscure a discoverability problem if supplied to a usability-test participant.

64. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    The human-AI interaction loop is cyclical: interfaces that encourage rubber-stamping can turn model-influenced approvals into misleading evaluation and training labels.

65. W3C WAI: Introduction to Web Accessibility

    Web accessibility means designing and developing websites, tools, and technologies so people with disabilities can perceive, understand, navigate, interact, and contribute. It includes auditory, cognitive, neurological, physical, speech, and visual disabilities. Accessibility therefore concerns participation in the interaction, not only the visibility of its final output.

66. Web Content Accessibility Guidelines 2.2

    WCAG requires keyboard operation subject to its stated exception, ways out of keyboard focus traps, and adjustable time limits subject to specified exceptions. Controls need programmatically available names, roles, and states. Status messages must be available to assistive technology without taking focus. For specified consequential submissions, error prevention can be supplied through reversibility, checking with correction, or review and confirmation. Previously supplied information required again in the same process should be populated or selectable, with exceptions. Conformance covers full pages and complete processes.

67. W3C WAI: What's New in WCAG 2.2

    WCAG 2.2's Level AA focus requirement says a component receiving keyboard focus must not be entirely hidden by author-created content, subject to the stated notes. Its Level AA dragging requirement calls for a single-pointer alternative that does not require dragging, unless dragging is essential or the unmodified user agent determines the functionality. W3C illustrates list reordering with clickable up and down controls. These requirements address different barriers: finding the focused control and operating a task without precise dragging.

68. Accessible Rich Internet Applications 1.2

    Live regions expose changing content to assistive technologies. Polite updates normally wait for a suitable opportunity; assertive updates can interrupt and clear queued speech, so the specification discourages them unless interruption is imperative. aria-busy can mark a region while multiple changes are made, allowing assistive technology to defer them and process a completed update together. These mechanisms support grouping updates instead of treating every text fragment as an urgent announcement.

69. WAI-ARIA Authoring Practices: Developing a Keyboard Interface

    Keyboard focus identifies the active interaction target and is distinct from selection. Focus should remain visible and move predictably. Removing a focused item or closing a dialog requires deliberate placement on a logical remaining control; otherwise focus can fall back to the document body. Automatically selecting content whenever focus moves can become particularly burdensome when each selection triggers a network request.

70. Understanding Success Criterion 1.4.1: Use of Color

    Color must not be the only visual means of communicating information, an action, or a requested response. Text or another visible distinction can supplement it. W3C explicitly includes errors, required fields, and successful database updates among relevant examples. Supplying the meaning only to assistive technology is insufficient for sighted people who cannot distinguish the colors; they also need a visible alternative.

71. Understanding Success Criterion 1.4.10: Reflow

    Reflow lets enlarged content fit the available viewport without losing information or functionality or requiring scrolling in two dimensions. For vertically scrolling content, the Level AA criterion specifies a width equivalent to 320 CSS pixels. Content requiring a two-dimensional layout, such as a data table, has an exception, but that exception does not automatically cover surrounding controls or the text inside individual cells. Repeated sideways scrolling to read lines increases effort and can make users lose their place.

72. Evaluating Web Accessibility Overview

    Accessibility should be evaluated early and throughout development. Automated evaluation tools assist the work, but no tool alone can establish that a site meets accessibility standards; knowledgeable human evaluation is required.

73. Involving Users in Evaluating Web Accessibility

    W3C recommends involving people with disabilities throughout development, including completing sample tasks on prototypes. Their participation reveals usability problems that conformance inspection alone can miss. User testing should be combined with standards evaluation; neither a few participants nor one disability group represents all accessibility needs. Reports should state methods, participant characteristics, and the scope of conclusions.

74. NIST AI RMF Core

    NIST connects measurement to deciding whether a system achieves its intended purpose and whether development or deployment should proceed. Risk responses include mitigation, avoidance and acceptance; remaining risks should be documented. The framework includes considering viable non-AI alternatives and assigning responsibility for superseding, disengaging or deactivating systems whose outcomes conflict with intended use. It also calls for evaluating measurement processes themselves and maintaining post-deployment monitoring, feedback, incident response, recovery and change management.

75. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    Lineage must survive graph mutation: entity merges retain both source sets, and invalidation records the new evidence responsible for the change.

76. Breaking the Chain: Agent Continuations for Resumable AI Workflows

    The speaker reports implementing suspension triggers based on elapsed time, turn count, and asynchronous requests, extending the continuation mechanism beyond approval gates.

77. LangSmith: Feedback data format

    LangSmith stores feedback separately from execution records and associates it with a trace or intermediate run. Sources include application feedback, human annotation and offline or online evaluators. The documented record includes run identity, creation and modification timestamps, a criterion key, numerical or categorical judgment, comments, correction information and feedback-source details. Source metadata and user identity can accompany the judgment.

78. The era of unbounded products: Designing for Multimodal I/O

    Highlight what matters, establish hierarchy, and reuse familiar interaction structures to make an unbounded product understandable.

79. Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

    The speaker reports that revised proctoring guidelines improved rejection of fabricated alerts by explicitly requiring independent video evidence.

80. UX Design Principles for (Semi) Autonomous Multi-Agent Systems

    User-facing observability should stream activity and application updates while exposing timing and token usage for debugging.

81. UX Design Principles for (Semi) Autonomous Multi-Agent Systems

    Cost-aware delegation requires inspecting proposed actions and estimating their risk or cost before deciding whether human involvement is needed.