Computer Use

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. CSSOM View: coordinate origins, scrolling, and zoom

   CSSOM coordinates generally use CSS pixels. clientX/clientY are viewport-relative; page coordinates are document-relative, with pageX = clientX + scrollX and the analogous y relation for the same view. screenX/screenY use the web-exposed screen origin. The visual viewport can pan within the layout viewport; offsetLeft/offsetTop describe that displacement. Page zoom and visual-viewport scale are distinct; devicePixelRatio includes page zoom but excludes pinch scaling. Derived mapping: for an axis-aligned screenshot crop, x_input = origin_x + x_image/k_x, where k_x is image pixels per input unit; likewise for y. Preserve crop origin, resizing, viewport offsets, scroll position, and scale. Desktop screenshots additionally require the content area's position within the screen.

2. The agent-ready web: Simplify user actions with WebMCP

   Layout changes between inspection and clicking can invalidate an agent's chosen coordinates.

3. Computer use tool — Claude Platform Docs

   A computer-use system runs an observe–act loop: the model requests actions, the application executes them in its environment, and tool results return to the model. Screenshots expose resulting state. Coordinates refer to the screenshot’s pixel space, so resizing or display scale requires a corresponding coordinate mapping. The documentation recommends a dedicated environment with minimal privileges, restricted network access, and human confirmation for consequential actions. It also warns that coordinate errors, unexpected actions, and prompt injection remain possible.

4. The Current State of Browser Agents

   Most browser agents described here repeatedly observe browser state, reason about the next step, and act before observing again.

5. Introduction to desktop flows

   Robotic process automation, or RPA, automates repetitive, rule-based software work rather than controlling a physical robot. Power Automate desktop flows can be assembled from predefined actions or recorded interactions and run later. Microsoft describes workflows spanning files, email, spreadsheets, websites, legacy applications, and terminal emulators. Targeting can use interface elements, images, or coordinates. A representative example extracts website data and places it into an Excel file.

6. Playwright: API testing and shared authentication state

   browserContext.request and page.request share the browser context's cookie storage: outgoing requests obtain its cookies, and response Set-Cookie headers update that storage. A separately created APIRequestContext has isolated cookies. Exporting storageState and initializing another context transfers authentication state; it is not a live synchronization link. The examples create an issue through an API, then navigate and assert its appearance, or submit through the UI and fetch the resulting issue by ID. These demonstrate cross-surface postcondition checks. Engineering implication: after switching surfaces, obtain fresh rendered state or a fresh API response; cookie sharing does not refresh an existing DOM, screenshot, or previously fetched response.

7. Computer-use models will agentify the web, not APIs

   An agent can test a discount by executing the customer workflow and checking the resulting cart price against the stated offer.

8. Can Oncology Workflows Run Without Human Touch? - Anant Shankhdhar, Risa Labs

   Portal automations remain fragile at runtime; the speaker describes a self-healing loop to detect and mitigate production breakages.

9. From RL to IRL — Gaurav Mishra, Amazon AGI Lab

   Neither DOM access nor screenshots alone guarantee enough context to distinguish the intended action from distracting or adversarial content.

10. Playwright: Navigations

    Hydration is the stage in which JavaScript attaches interactive behavior to an initially rendered page. Playwright documents a failure where an enabled-looking button receives a click before its event listener exists, so nothing happens; text entered into an early-rendered input can similarly disappear when hydration finishes. A page can therefore look actionable before its application behavior is ready. The documented application-side remedy is to keep interactive controls disabled until hydration completes. Browser load events also do not provide a universal boundary after which all application activity has finished.

11. Core Accessibility API Mappings 1.1

    An accessibility tree is a hierarchy of accessible objects exposing control roles, names, values, transient states, relationships, events, and supported actions to assistive software. The browser's DOM instead represents document elements and their structure and state. These are parallel representations rather than identical trees: elements without relevant semantics may be omitted from the accessibility tree. Native desktop applications also expose accessibility information through platform APIs. WAI-ARIA metadata generally changes exposed semantics without changing visual rendering or browser behavior, so semantic information and appearance need not agree.

12. Developing a computer use model — Anthropic

    Anthropic’s original computer-use report connects visual understanding with action selection: the model interprets a screen image, reasons about the next operation, and generates mouse or keyboard commands through tools. Locating a target requires grounding a semantic intention in screen coordinates. The report describes pixel-coordinate training and observed self-correction. This separates three abilities that can fail independently: understanding the task, recognizing the relevant visible element, and executing the right operation in the current interface state.

13. Playwright: Locators

    A semantic locator describes a target by user-facing attributes such as role and accessible name, rather than by one remembered screen coordinate. Playwright resolves the current DOM element each time the locator is used, so re-rendering can replace the underlying element without invalidating the intended selector. Operations that require one target are strict and fail if several elements match. Scoping a locator to a relevant container resolves ambiguity more meaningfully than blindly choosing the first repeated label.

14. Eager: Programming Repetitive Tasks by Demonstration

    Allen Cypher’s Eager inferred repetitive procedures from demonstrated actions rather than merely replaying recorded coordinates. Its implementation summary dates the system to 1990. Application knowledge distinguished objects by names, numbers, or identifiers; anticipation highlighting showed the next inferred action before users handed over execution. In a seven-person study involving three tasks each, Eager detected patterns in all 17 tasks participants performed correctly. All participants nevertheless expressed discomfort relinquishing control, prompting additions of backup copies and stepwise confirmation. The system could not infer conditionals or nested loops, and highlighting individual actions did not adequately communicate termination conditions.

15. Microsoft Active Accessibility

    Microsoft’s August 2001 account states that developers had used Active Accessibility since 1997. Its purpose was interoperability between applications and accessibility aids, including screen readers and voice-input utilities. Instead of requiring those aids to infer everything from appearance, applications exposed objects’ types, names, locations, and current states, together with navigation and change notifications. Standard controls received built-in support; custom controls needed suitable information exposure. This established a semantic interface to graphical software alongside its rendered appearance.

16. Sikuli: Using GUI Screenshots for Search and Automation

    Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller’s MIT work, presented at UIST 2009, made screenshot fragments usable as targets in automation scripts. Sikuli searched the current screen for an image pattern, obtained its location, and directed mouse or keyboard actions there. This addressed limitations of fixed-coordinate macros and applications lacking convenient automation APIs. Small patterns used template matching; larger patterns could use invariant visual features. Published examples include navigating maps and turning a cross-application screenshot tutorial into an executable visual script.

17. Multimodal Web Navigation with Instruction-Finetuned Foundation Models

    Furuta and colleagues’ WebGUM combined an instruction-finetuned language model with a vision encoder, jointly trained offline on navigation demonstrations. Screenshots, HTML, action history, and the instruction became inputs to a model producing click or type actions. Recent screenshots supplied temporal information, while image patches supplied local visual information. The ICLR 2024 paper reports 94.2% average success for its multimodal configuration versus 88.7% for its HTML-only configuration across 56 MiniWoB++ tasks, with 100 evaluation episodes per task. This illustrates a route from specialized interface policies toward adapting pretrained multimodal models.

18. World of Bits: An Open-Domain Platform for Web-Based Agents

    Shi, Karpathy, Fan, Hernandez, and Liang introduced World of Bits at ICML 2017 to make web interfaces environments for learning agents. Observations combined rendered pixels and DOM information; actions used keyboard and pointer events. MiniWoB supplied 100 small tasks with authored rewards. FormWoB recorded and replayed HTTP traffic from four flight-search interfaces, avoiding repeated live requests and reducing website-change variability. Trained models sometimes generalized between queries from the same template, but substantial errors remained. The work made both interface interaction and reproducible task construction explicit research problems.

19. Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

    Liu, Guu, Pasupat, Shi, and Liang’s ICLR 2018 work addressed sparse rewards and unsuccessful exploration in web tasks. Demonstrations supplied high-level workflow constraints that guided exploration, while successful interactions trained a separate DOM-based policy. Their MiniWoB++ additions introduced longer procedures, natural-language variation, and changing layouts. In the reported experiments, workflow-guided exploration reduced failures such as submitting before all requested checkboxes were selected or repeatedly checking and unchecking one box. DOMNET represented both spatial proximity and relationships in the document tree.

20. The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans

    Turn a discovered interaction sequence into executable code, while retaining model calls for variable content or judgment.

21. Playwright: Auto-waiting

    Before clicking, Playwright checks that the locator identifies one element and that it is visible, stable, enabled, and able to receive pointer events. It waits for these conditions within a timeout, reducing races with rendering and overlays. Stability is defined by a bounding box unchanged across consecutive animation frames. Assertions can separately retry until an expected condition appears. These mechanisms distinguish readiness to dispatch an action from observation that the application reached the desired postcondition.

22. Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

    The Terminus agent from Terminal Bench illustrates a more granular action space: a Tmux stream with character-level input and output.

23. Playwright: Actions

    Playwright exposes both control-addressed operations and lower-level pointer inputs. fill focuses an editable element and triggers an input event; pressSequentially sends character-by-character keyboard events; press focuses a selected element and produces a keystroke. These operations therefore exercise different input paths. Dragging comprises targeting the source, holding a mouse button, moving, and releasing at the destination. The scrolling example first hovers the intended scrolling container before issuing wheel input, or explicitly changes that element's scroll position. Programmatically dispatching a click bypasses the real interaction conditions that ordinary pointer actions check.

24. Computer-use models will agentify the web, not APIs

    Combine visual observation with code execution: use whichever action mechanism fits the task, then inspect the rendered result.

25. Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

    Waiting for a complete tool response implicitly discretizes observation and action, simplifying reasoning while limiting real-time reactions.

26. UI Events: focus contexts and keyboard targets

    Focus has three layers: the operating system selects an application; the browser selects browser chrome or a document; the document selects a focusable element. A document can retain its focused element while another application receives input. Within a document, keyboard events target the focused element, falling back to body or the root element when necessary. Tab can move focus between keydown and keyup, so even one keystroke can have different event targets. Enter or Space can activate a focused control rather than insert text.

27. WAI-ARIA Authoring Practices: Dialog (Modal) Pattern

    A modal dialog overlays another interface and makes content outside it unavailable for interaction. In the documented pattern, opening the dialog moves focus inside it, and Tab cycles among its controls. Closing generally returns focus to the invoking element, but workflow changes or removal of that element can require a different destination. For difficult-to-reverse operations, the guidance recommends considering initial focus on the least destructive action. Setting aria-modal communicates semantics to assistive technology; application code must actually enforce the corresponding interaction behavior.

28. WebDriver: contexts, sessions, and input ordering

    WebDriver selects a top-level browsing context by window handle, then optionally a frame for subsequent commands. This selection is independent of OS window focus. Element Send Keys focuses its keyboard-interactable target; pointer actions use viewport CSS coordinates. An HTTP session owns current-context state, a request queue, and input state associated with top-level contexts. Endpoint nodes permit one HTTP session. Input sources retain pressed keys/buttons; multiple sources can execute within coordinated action ticks. Engineering inference: controllers sharing a session need one coordinator that serializes context selection, observation, input, and verification as a unit. Queuing individual commands cannot prevent another controller switching context between selection and typing.

29. From RL to IRL — Gaurav Mishra, Amazon AGI Lab

    Execution monitoring should detect unproductive behavior, while audit logs should preserve both actions and effects for later verification.

30. OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents

    Abhyankar, Qi, and Zhang’s June 2025 study measured Agent S2 on a 37-task OSWorld subset, using GPT-4.1 for planning, reflection, and retrieval, plus UI-TARS-7B-DPO for grounding on one NVIDIA A6000. With a 50-step limit, planning and reflection accounted for approximately 75–94% of elapsed task time across application-group averages. Screenshot capture and input execution contributed much less. The recorded loop separated retrieval, planning, grounding, execution, observation, and reflection, showing why an agent can dispatch individual inputs quickly yet complete a workflow slowly.

31. The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans

    Use a sense–act–verify loop with one action per iteration, and verify its effect through a separate observation channel.

32. Your Agent Didn’t Fail. Your Harness Did.

    Internal acceptance does not prove the intended result appeared at the user-visible boundary.

33. Viewing all events for ticket updates

    Zendesk exposes ticket events through the ticket interface, allowing an operator to inspect updates and associated notifications after acting. Updates appear separately, changed fields show previous and new values, and notification identifiers open previews of the corresponding emails. Event information can include organization, requester, CCs, status, and triggers. This offers an application-visible readback path for checking which ticket changed and which notification was generated, without assuming the agent has direct database access.

34. Troubleshooting email deliverability

    Zendesk distinguishes an agent's ticket comment from successful outbound email communication. Its troubleshooting procedure checks delivery-failure indicators beside recipients and inspects ticket events to determine whether a trigger actually sent an email under the comment. A customer may receive no email because no notification trigger ran; other delivery problems can occur even when a trigger sends it. The ticket conversation, notification event, and delivery evidence therefore answer different verification questions.

35. Caching UI Automation Properties and Control Patterns

    Windows UI Automation can fetch selected properties and control patterns in bulk, reducing cross-process calls. A cache request defines which elements and properties are included; omitted properties are not implicitly available. Cached information remains valid only while the UI is unchanged. Clients must obtain another snapshot, commonly in response to UI events. Building an updated cache does not update existing element references. A cached-only reference cannot retrieve current properties or invoke actions on the control. Thus, retaining a semantic snapshot does not retain a live, actionable view of the application.

36. From RL to IRL — Gaurav Mishra, Amazon AGI Lab

    The talk's 'flight school, not just exams' approach trains recovery inside messy simulations instead of silently resetting failed runs.

37. Making retries safe with idempotent APIs

    An idempotent retry repeats one logical operation without an additional effect. AWS describes deduplication keyed by caller identity and a caller-provided request identifier. Identical parameters alone cannot establish identity because two identical resource creations may be intentional. Store original parameters with the identifier and reject mismatched reuse. Recording the token and performing mutations must be atomic, avoiding effects without deduplication records or records without effects. Retries reuse the original identifier and receive semantically equivalent responses within the service's retention contract. Late requests require retention beyond resource deletion. Derived race: an absence/status read cannot prove a timed-out creation will never commit if its request remains in flight; a fresh identifier can produce a duplicate.

38. Temporal Activity Execution

    An Activity Execution can comprise multiple task attempts. Temporal relies on timeouts to detect lost work, including worker crashes after invocation, and retries according to policy; limiting attempts to one prevents retry but does not prove an external effect failed. Cancellation is cooperative: activities receive service cancellation through heartbeats, can ignore it, and workflows may proceed without waiting for acceptance. A timed-out attempt may therefore continue while another attempt runs. Application consequence: treat an unconfirmed external mutation as uncertain, retain its operation identifier, reconcile against the receiving system, and use enforced idempotency or explicit recovery before repeating it. Timeout or cancellation is not evidence that a payment, message, or write was reversed.

39. Your Agent Didn’t Fail. Your Harness Did.

    Trace one real run from trigger identity through inherited state, authority, execution attempts, and surviving external evidence.

40. Compensating Transaction pattern

    Compensation performs new application-specific actions to counter already completed work. Unlike transactional rollback or restoring an old snapshot, it must accommodate concurrent changes and may produce a different final state. For example, canceling a reservation can incur charges rather than restore the original balance. Record completed steps, affected resources, undo information, and compensation progress durably. Correlate original and compensating operations for auditing. Compensation need not run in strict reverse order; some steps can run concurrently. Because compensation itself can fail, steps should be idempotent and resumable. Timeouts may detect blocked work, while unrecoverable failures require alerts and potentially manual intervention.

41. OWASP Access Control

    Authentication establishes identity; authorization decides which actions that identity may perform on particular resources. A user allowed to initiate a transfer must still be authorized for the source account. Least privilege limits the authority of running code and service accounts, while centralized checks reduce inconsistent enforcement. In an AI application, tool availability and a model-produced argument are therefore insufficient grounds to execute a business operation; the application must apply resource- and action-level policy.

42. The Protection of Information in Computer Systems

    Saltzer and Schroeder describe complete mediation, fail-safe defaults and least privilege: check authority for accesses, deny absent permission, and limit a component’s granted powers. These principles apply during recovery as well as ordinary execution. In an agent loop, tool proposals and retrieved instructions must therefore pass an independently enforced authorization boundary before they can affect protected resources. Model capability does not establish the caller’s authority.

43. Control access to screen and system audio recording on Mac

    macOS provides per-application controls over screen and system-audio recording. Users can inspect and change these permissions in Privacy & Security settings. Apple warns that information collected by third-party applications is governed by their own terms and privacy policies. Permission to capture an observation is therefore a separately administered capability from accessibility-based computer control.

44. Allow accessibility apps to access your Mac

    macOS requires explicit permission for third-party applications that use accessibility features to access and control the Mac. Users manage and revoke this access per application in Privacy & Security settings. Apple advises granting it only to trusted applications and notes that third-party handling of accessed information is governed by those applications’ terms and privacy policies.

45. Sign in to multiple accounts at once

    Google permits several accounts to remain signed in simultaneously within a browser, with account switching through the application’s profile menu. When Google cannot determine the intended account, it may use the default account, which is often the first account signed in. The browser session alone therefore does not uniquely identify the account governing a particular operation. For computer use, checking the application’s displayed account is a necessary contextual check rather than assuming a familiar browser profile implies the intended identity.

46. Playwright MCP

    Playwright MCP supports persistent profiles that retain login state, isolated sessions that discard session state when closed, and an extension that connects to existing browser tabs and their authenticated state. These modes expose materially different starting authority. Its documentation explicitly says the server is not a security boundary. Allowed and blocked origins do not constrain redirects; secret substitution is a convenience rather than a security mechanism, and workspace file restrictions are described as guardrails rather than enforceable isolation.

47. OWASP Logging Cheat Sheet

    OWASP advises against directly logging passwords, access tokens, session identifiers, connection strings, encryption keys, sensitive personal information, and payment data; remove or appropriately protect sensitive fields. Validate and sanitize event data crossing trust boundaries to prevent log injection. Restrict and periodically review read access, record access to logs, protect transfer and storage, and apply retention and disposal rules to debug logs, backups and extracts as well as primary logs. Applied to agents, collect the identifiers, timings, outcomes and selected diagnostic fields needed for investigation; do not assume entire prompts, retrieved documents or tool responses are safe to retain.

48. NIST Privacy Framework 1.0: lifecycle and minimized audit evidence

    The framework inventories data elements, processing purposes, actions, owners and flows. Policies define permitted uses and retention periods; the data lifecycle aligns with system development and operations. Authorizations must be maintained and revocable, access limited by least privilege, and deletion and destruction performed under policy. Audit records themselves must incorporate data minimization. Engineering application: define the decision evidence needed for review, its purpose, authorized readers, retention trigger and disposal method before logging. Retain the necessary decision, model and policy versions and relevant evidence without indiscriminately copying personal data into logs, prompts or backups. Where review requires sensitive evidence, constrain fields, access and retention rather than treating auditability as permission to keep everything. Assess removal and disclosure across downstream copies and service providers.

49. From RL to IRL — Gaurav Mishra, Amazon AGI Lab

    Calibrated confidence should account for action risk and authority; maximizing autonomy is not always the right objective.

50. Writing drafts of public replies in tickets

    Zendesk Agent Workspace provides an explicit draft mode for public replies. With draft mode enabled, submitting displays a warning; the operator can return to editing or choose Send to make the reply public. Turning draft mode off before submission removes that warning. The setting normally follows the operator across tickets, with different behavior when an administrator enables drafts by default. In chat and messaging draft mode, Enter neither sends the message nor opens the warning. This supplies a concrete example in which the consequence of an input depends on channel and current UI mode.

51. OWASP Transaction Authorization Cheat Sheet

    Transaction authorization should let the user acknowledge significant transaction data, such as destination and amount, and bind approval to that transaction rather than an unrestricted session. The server controls authorization data and permitted state transitions. Changes to transaction data invalidate prior authorization or restart the process. Credentials should be unique per operation and valid only for a limited interval. A final server-side gate immediately tied to execution verifies that the transaction was properly authorized, preventing skipped checks and substitution between approval and use. For an agent, approving a draft action therefore must not silently authorize a changed target, payload, or scope.

52. Building Your Own Secure AI Workflows: Human-in-the-Loop Automation with n8n

    Place approval in the execution path so the agent cannot bypass it by choosing to call the underlying tool directly.

53. The Protection of Information in Computer Systems: Basic Principles

    Least privilege limits each user and program to permissions needed for its job. Complete mediation requires authority checks on every access to every object, including initialization, recovery, shutdown, and maintenance. It requires reliable identification of request sources and care with cached authorization when permissions change. Fail-safe defaults base access on explicit permission. Applied to a harness, these principles imply that protected operations must pass through an enforcement mechanism that the requesting program cannot bypass or modify; a prompt instructing the model to behave is not that mechanism.

54. Computer-Use 2.0

    Cua’s July 13, 2026 developer account dates its open-source Driver release to April 22. The driver targets windows rather than treating the entire desktop as one input destination, returning a window’s screenshot and accessibility tree together. Its action ladder tries accessibility operations, coordinate input, and finally temporary foreground activation. Sessions associate actions and recordings with assigned windows. The team describes regression tests that independently check target-state changes and verify that unsupported routes refuse without stealing focus, rearranging windows, or moving the physical cursor.

55. Mitigating the risk of prompt injections in browser use — Anthropic

    A browser agent reads untrusted pages while operating within an environment that may hold authenticated data and action permissions. Attackers can place instructions in page text, images, or deceptive interface elements to redirect the agent away from the user’s task. Anthropic describes layered mitigations including adversarial training and classifiers that inspect untrusted content. The mechanism is an authority-confusion risk: content to be read can be misinterpreted as instructions to obey. Restricting access and requiring approval for consequential actions limits the impact of a mistaken decision.

56. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    Indirect prompt injection places attacker instructions in material an application is likely to retrieve, rather than in a direct user request. The paper demonstrates how applications can confuse external data with instructions, allowing retrieved text to redirect behavior or influence subsequent API calls. This makes the provenance and trust level of a context item separate from its relevance to a query: useful retrieved material can still be adversarial.

57. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    Indirect prompt injection places attacker-authored instructions in material an application retrieves at inference time, such as web pages or messages. When inserted into model context, that material can be interpreted as instructions and redirect subsequent answers or API use without the attacker controlling the user's request or model weights. The paper demonstrates attacks on integrated systems and develops a taxonomy including information theft and manipulated content. Engineering implication: retrieval and tool results remain untrusted data; the application must independently enforce resource access, tool permissions, transaction approval, and data-release restrictions. A model's interpretation of retrieved text cannot grant these privileges.

58. From Arc to Dia: Lessons learned in building AI Browser

    The 'lethal trifecta' combines private-data access, exposure to untrusted content, and external communication in one assistant.

59. From Arc to Dia: Lessons learned in building AI Browser

    The speaker warns that tagging untrusted content and separating instructions from content can help but do not eliminate prompt injection.

60. Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

    An environment needs an instruction, a sandbox in which to act, and a verifier that assesses completion under a stopping condition.

61. The Current State of Browser Agents

    A useful evaluation needs realistic, feasible tasks, an explicit assessment method, and consideration of the execution infrastructure.

62. From RL to IRL — Gaurav Mishra, Amazon AGI Lab

    A process reward model should penalize dangerous intermediate actions even when the requested outcome is achieved.

63. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

    SeeClick defines GUI grounding as locating an interface element from a screenshot and language instruction. ScreenSpot pairs screenshots and instructions with annotated actionable-element bounding boxes across mobile, desktop, and web interfaces, including text controls and icons. Its click-accuracy metric counts predictions whose point falls inside the annotated target box; when a model predicts a box, its center is used. This evaluates target localization on supplied images. The paper separately evaluates downstream interaction tasks, demonstrating that grounding evaluation and task evaluation have different units and success criteria.

64. WebArena: A Realistic Web Environment for Building Autonomous Agents

    WebArena provides self-hosted, interactive web applications and natural-language tasks with functional validators. Evaluation checks resulting application content, allowing different valid action paths rather than requiring a reference click sequence. The paper’s failure analysis includes agents using an easily visible but irrelevant report and repeatedly entering text despite observations showing it already present. These examples separate observation availability from correct state interpretation. They motivate preserving task constraints, checking progress, and stopping unproductive action loops.

65. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

    OSWorld defines tasks in real operating-system environments with controlled initial states and executable evaluators. Evaluators inspect resulting artifacts, settings, or application data to judge functional completion rather than merely checking whether the agent produced a plausible action sequence. The paper’s analysis identifies errors in screenshot grounding, repetitive actions, unexpected windows, and application knowledge. Task setup and evaluation configuration are therefore part of the measurement, and long trajectories can fail even when many individual clicks are reasonable.

66. Computer-Use 2.0: Agents Just Got Multi-Cursor

    Cua-Bench separates task initialization, a reference GUI trajectory, and environment-based outcome evaluation.

67. The Protection of Information in Computer Systems: Basic Principles

    Least privilege gives each program or user only the authority needed for its job, reducing damage from mistakes and the interactions requiring audit. Separation of privilege requires multiple conditions or independently held keys for protected access. Complete mediation checks authority for every access. Applied to verification, use a distinct principal with narrowly scoped reads of authoritative evidence and no execution, mutation or permission-granting authority. If an action requires independent approval, enforce both authorities at the protected operation.

68. D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies

    Chen and colleagues’ November 2025 D-GARA preprint evaluates Android agents while injecting interruptions during execution. Rules inspect interface XML to trigger permission dialogs, alerts, or other disturbances; the agent’s response can redirect it into a different trajectory rather than returning it to a prescribed next screenshot. After actions, a validator checks specified UI properties against the goal, with human review of automatically validated results. The reported experiments found degraded task performance under interruptions. This tests recovery from encountered states rather than only predicting actions on clean reference screenshots.

69. Computer Use at the Edge of the Statistical Precipice

    DigiWorld nests apps, scenarios (task templates), configurations and stochastic rollouts. Configurations combine instance parameters, data profiles, themes and initial UI states. Per-configuration success is the mean binary rollout outcome; the paper uses Wilson intervals. Suite performance averages per-app means equally, holding the curated apps fixed. Its hierarchical bootstrap resamples scenarios within each app, enabled environmental axes independently, then rollouts within configurations; interval endpoints are bootstrap quantiles. This targets variation within that fixed suite, not a sampled population of apps. The experiments report separate model intervals. Matched configuration pairs in Appendix H vary one environmental axis while holding others fixed; these are not paired agent-version comparisons. The paper does not specify a paired estimator or confidence interval for the difference between agent versions.

70. Computer Use at the Edge of the Statistical Precipice

    The speaker reports that blindly replaying successful task traces can match or outperform their source frontier model on deterministic benchmarks such as OSWorld or MobileWorld.

71. Computer Use at the Edge of the Statistical Precipice

    The speaker argues that pass@k on deterministic environments can encode the same weakness exposed by a replay agent.

72. The Current State of Browser Agents

    The reported benchmark found a larger performance deterioration on write tasks for autonomous agents than for the human-supervised baseline; production write workflows warrant rigorous internal evaluations.

73. Resolving an ambiguous payment request

    A timeout can leave the client unable to tell whether Stripe received or executed a request. Stripe documents retrying with the same key and parameters until a server result is obtained, using backoff. An HTTP 500 remains indeterminate: side effects may exist even though the cached response stays unchanged. Stripe may reconcile partial mutations and emit webhook events for resulting objects. Supplying a local operation identifier in metadata lets the application correlate these objects with its own pending operation. Engineering consequence: preserve pending state until authoritative provider evidence resolves it; do not infer failure solely from a timeout.

74. Your Agent Didn’t Fail. Your Harness Did.

    Approval must remain bound to one specific action and its scope, identity, arguments, and lifetime; expiration should terminate the approval path.

75. Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

    Agentic MapReduce can extract user corrections from many sessions, aggregate recurring failure categories, and use those categories to inform new evaluation tasks.

76. Computer-Use 2.0: Agents Just Got Multi-Cursor

    Cua Driver describes background interaction that first tries accessibility-based execution and falls back to pixel-based background clicking.