Contents
  1. Purpose and development
    1. Define the completed work
    2. Turning points in process automation
  2. Redesign and allocation
    1. Change the process before automating it
      1. What event records reveal
      2. Change dependencies deliberately
    2. Allocate work and authority separately
      1. Make rule conflicts visible
    3. Bound the dynamic decision
  3. State and coordination
    1. Name the authoritative records
    2. Accept events against current state
    3. Transfer responsibility explicitly
      1. Coordinate parallel obligations
  4. Human decisions and exceptions
    1. Make review a consequential decision
      1. Test the reviewer decision
    2. Operate the exception workload
      1. Capacity constrains automation
  5. Integration and recovery
    1. Preserve meaning across applications
    2. Recover from the effects that occurred
      1. Partial completion is still real
  6. Outcomes and daily operation
    1. Measure the complete process
      1. Interpret the comparison
    2. Keep the workflow usable and owned
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

Workflow Automation

Workflow automation coordinates work across people and applications. Its purpose is to make a useful result easier to obtain—not simply to make individual tasks run faster. The central design decision is which work to remove, which to implement with explicit rules, where learned interpretation helps, and where people should retain judgment or authority. Those choices become a dependable process only when handoffs, exceptions, recovery and operating responsibility are designed with them.

Purpose and development

Define the completed work

A business process connects activities that transform inputs into a result. Its workflow specifies how that work proceeds: which activities occur, what permits advancement and how people or software participate. Automating an activity changes how one part is performed; redesigning the workflow can also change which activities are needed and how they depend on one another.

A case is one instance of the work: one service request, invoice discrepancy or application. The reusable process describes how cases should be handled; a particular case has its own supporting records, completed work and remaining obligations. Changing the process definition and advancing that case are different operations. The Case Management Model and Notation specification formalizes this distinction between a case model and its instances.

Define completion from the recipient's perspective. Drafting a support reply produces an artifact; resolving the request requires satisfying the agreed service need. A disposition records how the case was handled, such as fulfillment, rejection or cancellation. These can all be legitimate outcomes, but they are not interchangeable. Even a closed status need not establish customer-confirmed resolution: Zendesk can close tickets through automation, after which a reply creates follow-up work rather than reopening the original.

Someone must maintain the complete arrangement, including obligations that cross teams. This process owner defines and maintains the process, as described in the ISO quality-management vocabulary; ownership does not automatically grant authority over every participating system or transaction. Before selecting automation, agree on the initiating condition, case identity, intended recipient, required result, allowed dispositions and downstream obligations. The customer acceptance agreement establishes these requirements; workflow design makes them operational.

Turning points in process automation

Process automation draws on several traditions because improving flow, changing the work itself, coordinating execution and allocating human responsibility are different problems. Their historical development explains why a more capable model does not make the older concerns disappear.

DevelopmentContribution
Toyota's stopping and flow traditions — 1896 and 1938 antecedentsToyota traces automatic stopping to Sakichi Toyoda's 1896 power loom and flow production at its Koromo plant to Kiichiro Toyoda in 1938. In the later Toyota Production System, jidoka means stopping on abnormality; Just-in-Time coordinates production with what is needed, when and in what quantity. These address quality and whole-process flow, not merely machine speed.
Business process redesign — June 1990Thomas Davenport and James Short's The New Industrial Engineering treated information technology and process redesign as mutually informing choices. Their cross-organizational perspective challenged departments optimizing their own tasks while nobody owned the complete result.
Workflow Reference Model — 29 November 1994David Hollingsworth and the Workflow Management Coalition's archived reference model separated process definition, execution control and interaction with people and applications. Common components and interfaces addressed incompatible workflow products. It explicitly distinguished redesigning a procedure from automating it unchanged.
Function allocation — May 2000Raja Parasuraman, Thomas Sheridan and Christopher Wickens's human–automation framework separated acquiring information, analyzing it, choosing actions and implementing them. Different functions could receive different degrees of automation.

Business process reengineering concerns redesigning the process itself, rather than simply accelerating its existing tasks. Business process management is the broader continuing work of defining, operating and improving processes. Neither requires every activity to be automated. Manufacturing flow principles also require adaptation to service work: a waiting request represents an obligation to someone, not inventory that can necessarily be discarded. These traditions supply complementary design questions, not a ladder toward eliminating people.

Redesign and allocation

Change the process before automating it

Start from observed work, not just the procedure manual. The perceived process, actual process and desired process can differ. Map activities, responsible roles, decision conditions, waiting, variants and correction. Learning from work as it happens explains discovery; here its findings become decisions about what to retain, remove or rearrange. AHRQ's workflow-mapping guidance recommends checking descriptions through observation and testing redesigns before expanding them.

A bottleneck is the constraint limiting flow through the complete process. It is not necessarily the task with the longest individual duration. A quick approval performed by one overloaded reviewer can hold up more work than a lengthy activity with ample capacity. Separate active work from waiting, then identify what prevents useful completions. Faster drafting can otherwise move the constraint into review. Maggie Appleton's collaborative-engineering account describes how alignment deferred until a pull request leaves teams reviewing already-built work whose purpose was never agreed.

What event records reveal

Process mining uses recorded events to investigate how work actually proceeds. Each event identifies an activity and its case; timestamps help reconstruct timing, while records of who or what performed the activity help locate responsibility. Grouping events by case reveals recorded paths. Choosing an order as the case gives a different view from choosing an order line or delivery. Wil van der Aalst and collaborators' Process Mining Manifesto, developed for the 2011 Business Process Management workshops and published in 2012 proceedings, distinguishes three uses: discovering a process model from events, checking recorded behavior against an existing model, and enhancing a model with event information. None recovers work that was never recorded. Missing phone calls, informal approvals and off-system correction therefore require observation alongside log analysis.

Change dependencies deliberately

Redesign in this order: remove work that serves no requirement, simplify retained work, improve inputs and reconsider dependencies. Consider a small example in which a coordinator copies a submitted request into a second register, obtains approval and only then checks availability. Suppose the register adds no required information and the availability check is read-only and independent of approval. The revised process can reuse the original record and perform that check earlier, while fulfillment still requires both approval and availability. Removing the copy and changing the dependency are separate decisions; neither relaxes the completion requirement.

Change dependencies, preserve obligations

Change dependencies, preserve obligationsThe old path serializes request, duplicate entry, approval, availability check and fulfillment. The revised path removes copying, forks the original request into approval and read-only availability, then requires both valid results before fulfillment.Before · copied entry and serialized prerequisitesRequestRequesterDuplicate entryCoordinatorApprovalApproverAvailability checkOperations · read-onlyFulfillOperationsAfter · original record reused, both obligations retainedRequestRequesterApprovalApproverAvailability checkOperations · read-onlyBoth holdExplicit joinFulfillOperationsValid approvalAcceptable availabilityOnly copying disappears. Approval is still required; a completed rejection does not satisfy the join.
This fictional request process removes duplicate entry and an unnecessary dependency. Approval and the read-only availability check can proceed independently; fulfillment still requires valid approval and acceptable availability. Edges express prerequisites, not elapsed time or measured savings.

The proposal must still be checked with the people operating dependent processes. Earlier checks might become stale before fulfillment; a supposedly duplicate entry might support another team's obligation. The test is whether the changed arrangement preserves those obligations with less unnecessary work, not whether its diagram contains fewer boxes.

Allocate work and authority separately

Function allocation assigns parts of the work to people or software. Parasuraman, Sheridan and Wickens's framework is useful because collecting information, interpreting it, selecting an action and executing it need not share an executor. Their proposed evaluation considers human performance, automation reliability and the consequences of incorrect actions. Automating collection can help while leaving a consequential choice with a person; automating execution can also be appropriate after another actor has settled that choice.

Ordinary software applies explicitly programmed rules. A model applies behavior fitted from examples, which can help interpret varied language or documents; Machine Learning Fundamentals develops that distinction. Interpretation does not confer authority. A model can propose a request category, deterministic software can check explicit eligibility conditions, and a person can resolve disputed intent. Keep mandatory conditions and validated business state outside the model's discretion.

Choose an arrangement from the properties of the work, not from an autonomy ranking.
Work propertyExecution choiceAcceptance and authority
No required result depends on the activityRemove itProcess owner confirms that no obligation is lost
Explicit rules and structured inputsDeterministic softwareValidate inputs and enforce the rule at the protected operation
Variable wording, but a checkable interpretationModel proposes structured informationValidate against sources; clarify material ambiguity before commitment
Next investigation depends on findingsBounded agent stageLimit available operations and effort; return findings to the decision owner
Disputed intent or unsettled tradeoffsHuman decision with software assistanceThe responsible person settles the choice; software still enforces applicable constraints
Rare work or costly maintenanceRetain manual executionCompare the full operating cost before automating

Input ambiguity and rule clarity determine what can be specified. Consequence, reversibility and verifiability determine how much independent checking or authorization is needed. Frequency and maintenance burden determine whether implementing the automation is worthwhile. Decision support assists a decision owner without taking over the decision. Here, straight-through processing means completing a case without human intervention; it describes a path through the process, not a quality guarantee.

Make rule conflicts visible

A decision table maps input conditions to outcomes. Overlap needs explicit semantics. Suppose example rules say “amount below 100 → routine” and “new supplier → review.” Both match a new supplier's amount of 50. In Decision Model and Notation, Unique prohibits overlapping rules; Any permits overlap only when outputs agree; First chooses the first match. First therefore makes row order consequential and can hide inconsistencies. A table makes policy executable, but cannot establish that the policy itself is appropriate.

Bound the dynamic decision

A prescribed workflow can contain branches, repetition and parallel operations. An agent adds discretion over subsequent actions within a delegated task. Always reading three named records is a prescribed procedure; choosing which record to inspect next from an unexpected finding introduces a dynamic decision. Agent Engineering explains this boundary. Start with the simplest sufficient arrangement: more discretion adds behavior to assess and can add latency and cost.

Use an agent where the investigation genuinely benefits from that choice, not merely because a model appears somewhere in the process. Michael Albada's Building Applications with AI Agents recommends simple chains and branches where sufficient, while treating difficult-to-maintain branching structures as a reason to consider fuller agency. This is an architectural tradeoff, not a universal threshold or measured guarantee that an agent will be easier to maintain.

Give the delegated stage a workflow-facing contract:

  • PurposeThe result it must return and the evidence needed to accept that result.
  • ReachPermitted records, operations and external effects; investigation permission need not include repair permission.
  • LimitsTime, attempts or other effort allowances enforced by the surrounding system.
  • Return conditionsCompletion, insufficient evidence, exhausted limits or a needed action outside its authority, with a named receiving owner.

The surrounding workflow still owns case disposition and mandatory gates. Bloomberg's described semi-agentic architecture, for example, makes policy and factuality checks non-optional rather than letting the agent choose whether to invoke them. Mandatory invocation and check effectiveness remain separate properties. The broader problem of limiting cumulative effects belongs in Constrain accumulated effects.

State and coordination

Name the authoritative records

A system of record is the designated authority for particular business facts. Authority can be divided: one application owns an address while another owns an order's fulfillment status. IBM's operational-data-store guidance treats this designation as a business decision and requires explicit overwrite rules between systems. Recall the agreed entities and meanings from Making business meaning explicit before assigning their writers.

Keep three kinds of state separate. A model proposal records a suggested change. Workflow state records orchestration progress, such as waiting for review. Business-record state records the accepted facts in the owning application. All three can be durable while disagreeing: a saved proposal is still only a proposal, and an execution marked finished may have stopped before the business requirement was met. A conversation can help reconstruct intent without becoming the authority for a calendar event, ticket status or payment.

For each case, preserve a mapping among its business identifier, execution identifiers and external record identifiers. A new execution attempt need not mean a new business case. Assign each mutable field a permitted writer, and define which observations justify changing it. For example, the workflow can own “awaiting confirmation,” while only a response or fresh read from the receiving system establishes its actual record state. This mapping gives later notifications and recovery work something stable to refer to.

A delayed proposal must be accepted against relevant current state, not merely the copy used to prepare it. HTTP's If-Match mechanism can make an update conditional on the current representation's version. That protection covers the representation identified by its validator, not every related business fact. Preserve state at acceptance explains concurrency enforcement. Reading first and writing later is insufficient if another writer can invalidate the checked conditions in between.

In this fictional case, saved proposals, workflow progress and application-owned facts describe different state. A delayed v1 proposal cannot overwrite the current v2 representation. The protected operation must also enforce authority.

Accept events against current state

A trigger is a request, notification or scheduled occurrence that starts or resumes consideration of work. Event correlation associates it with the correct case. Receiving a trigger does not itself admit a new case or authorize an operation. Scheduled and webhook-triggered agents are therefore workflow participants even when nobody has an active chat session open.

A transition changes workflow state; a guard is a condition that must hold before the transition is permitted. The SCXML state-machine specification separates the triggering event, guard and target state. Apply that separation to business work: a received document may satisfy a missing-information condition, but only a still-open, eligible case may advance.

In this example, a notification reports that information is available. Its event ID identifies the notification; the case ID identifies the continuing work.
Incoming notificationCurrent case stateDisposition
First receipt of event E1Waiting for the reported informationRead the information; advance only if the guard now holds
E1 delivered againIts notification was already handledDo not treat it as a second business request
A different, late eventCase cancelled or otherwise terminalApply the explicit reconsideration policy; do not silently reopen

Delivery order is not necessarily business order. Stripe's webhook documentation, for example, describes duplicate and out-of-order delivery, and event objects that do not change when the underlying resource changes. A fresh resource read can therefore answer a different question from an old event payload. Deduplicating a notification also does not prove that its downstream write occurred exactly once.

An update produced by the automation can itself generate another notification. IBM's synchronization example uses update-origin information to recognize already synchronized records and avoid update echoes. That addresses a different problem from repeated delivery of one event. Define planned waits, information deadlines and explicit reopening conditions as business policies; expiry can lead to escalation, rejection or cancellation, but should not acquire an accidental meaning from a worker timeout.

Transfer responsibility explicitly

A handoff transfers responsibility together with the information, access and authority needed to continue. Its work item should name the case, requested result, receiving role, required records, acceptance deadline, escalation owner and return conditions. Sending an email only requests attention. Receiving that message, accepting ownership, starting work and completing it are separate events.

OASIS's WS-HumanTask specification of 17 August 2010 gives this distinction an explicit lifecycle. Ready work is available to potential owners; claiming it makes it Reserved; starting makes it InProgress. Completion remains separate. Missing ownership requires administrative action, and start or completion deadlines can trigger escalation. These task states organize responsibility but do not prevent unrelated software from writing external records.

Specify who remains accountable until ownership is accepted. An operational example is PagerDuty's incident lifecycle: acknowledgment records that a responder has claimed an unresolved issue; without acknowledgment, escalation continues. An acknowledgment timeout can resume escalation. A notification-delivery receipt is consequently weaker than accepted responsibility, and accepted responsibility is weaker than a resolved incident.

Delivery is not accepted responsibility

Delivery is not accepted responsibilitySending owner remains accountable through delivery until claim. Recipient owns the task from claim through completion; starting is a separate event. An alternative unclaimed deadline triggers escalation while work remains outstanding.Event order; spacing is not elapsed timeDispatchDeliveredClaimStartCompleteSending ownerRecipientAccountable until claimAccountable after claimReady · availableReservedInProgressAlternative: no claim by the acceptance deadlineDelivered, still unclaimedSending owner remains accountableEscalate outstanding workNot task completion; no automatic releaseDeadline expiresCompleting this work item may leave broader case obligations outstanding.
Example policy: the sending owner remains accountable until the recipient claims the work. An unclaimed deadline triggers escalation. Ready, Reserved and InProgress use WS-HumanTask meanings; the accountability policy is not a claim that WS-HumanTask and PagerDuty share an implementation. Task ownership does not fence external writers.

Coordinate parallel obligations

Swimlanes group work by responsible role. Business Process Model and Notation, or BPMN, provides more precise distinctions: pools represent participants, lanes categorize activities within a process, sequence flows express order and message flows represent communication between participants. Those distinctions prevent one arrow from ambiguously meaning both “must finish first” and “sends information to.”

Parallel obligations need a join condition. If fulfillment requires both approval and an availability check, finishing either alone is insufficient. SCXML similarly completes an enclosing parallel state only when all child regions reach final states. In a business workflow, additionally define which results count as acceptable: rejection or cancellation of one branch must not masquerade as successful fulfillment. Demonstrating operating ownership extends this responsibility transfer to the people who will operate the whole system.

Human decisions and exceptions

Make review a consequential decision

Human review is deliberate assessment before accepting a result or performing an action. Human-in-the-loop means a person participates at a defined point in the process, but that phrase alone leaves their job unclear. Review assesses a proposal; clarification supplies missing intent; authorization grants permission; takeover accepts responsibility for unresolved work. Request the intervention actually needed, as developed in Review before consequential commitment.

Place review where consequence, uncertainty, irreversibility or required authority justifies it. Give the reviewer the relevant source information, exact target and proposed change, applicable rule, unresolved issue and allowed dispositions. Approve, reject and request changes perform different work. A reviewer who cannot inspect the source, understand the domain or spend enough time is not an effective control merely because a button must be clicked.

Enforce approval on the execution path. The n8n workshop demonstrates a configured layer that intercepts the ordinary tool invocation before execution, rather than relying on the agent to choose a separate review tool. The distinction is architectural: required review must not be bypassable through another route to the same effect.

Bind authorization to the significant action data and its validity conditions. OWASP's transaction-authorization guidance requires a final server-side gate and invalidation when transaction data change. Applied here, approval of one recipient, payload and scope must not silently authorize another. A revised proposal may need new approval; a delayed proposal needs its relevant state and authorization rechecked. Recording an earlier yes-click is not enough.

Test the reviewer decision

Automation bias is inappropriate reliance on automated advice. Test whether reviewers challenge wrong assistance, not just how often they approve it. In When Machines Mislead, researchers inserted fabricated cheating alerts into previously certified test sessions. Revised guidance requiring independent suspicious behavior was associated with model-estimated rejection of fabricated alerts rising from 50% to 71%; rejection of genuine operational alerts also rose. Those genuine alerts were not ground-truth misconduct, so rejecting them cannot automatically be counted as an error. The studies also used different sessions and periods, not simultaneous random assignment. To evaluate review quality, distinguish whether reviewers accept or reject advice from whether their decisions are correct, using independently assessed cases where possible.

Operate the exception workload

An exception requires handling outside the normal path. An exception queue is tracked work awaiting that handling. A planned wait for an expected document is not automatically an exception; an expired deadline, conflicting information or an unavailable application may require one. Route by what can remedy the problem, not by a single generic error flag.

UiPath's business/application distinction is a useful starting point; ambiguity, policy disposition and uncertain external effects require additional treatment.
ConditionUseful responseReturn or exit condition
Missing or invalid informationRequest correction from the appropriate sourceRequired information becomes usable; unchanged input is not repaired by retrying
Technical application failureInvestigate or retry under the operation's policyDependency recovers and repetition is justified
Ambiguous intentAsk the responsible person to clarifyThe intended result and target are explicit
Known policy rejectionRecord the reason and notify the appropriate partyTerminate normally unless an authorized exception process applies
External effect unconfirmedAssign reconciliation workAuthoritative evidence resolves the effect, or an owner accepts further investigation

Each queue item needs an owner, reason, supporting context, priority, deadline and permitted disposition. Queue age is time since entry; backlog is outstanding work. A worklist without those meanings can hide abandoned cases. UiPath's review worklist illustrates assigned items with distinct review and retry dispositions. A recorded disposition still needs to mean something concrete about the underlying business problem.

Capacity constrains automation

Compare arriving handling effort with the effort reviewers can supply over the same period. Case counts alone are insufficient when difficult cases take much longer. If required work persistently exceeds capacity, the backlog grows even when the automated stages are fast. Amazon's queue-backlog guidance describes prioritization, isolation and backpressure—reducing accepted upstream work when consumers fall behind. Applied to human work, pacing must respect service obligations; urgent or old cases cannot simply be discarded.

Case counts are not handling capacity

Fictional five-day effort accounting: equal case counts can require different handling effort. Daily input is held constant, and unfinished effort carries forward.

Incoming effort: 300 min/day. Available effort: 240 min/day.
Outstanding after five days: 300 min (5 hours).
Outstanding effort, not individual waiting timeInitial outstanding effort followed by five end-of-day values. The line accounts for incoming effort minus handled effort; remaining work is never discarded.Outstanding effort (minutes)0150300InitialDay 1Day 2Day 3Day 4Day 5Days 1–5: end-of-day backlog
DayStart (min)Incoming (min)Handled (min)Remaining (min)
1030024060
260300240120
3120300240180
4180300240240
5240300240300

End-of-day backlog = max(0, previous backlog + incoming minutes − available minutes). Capacity beyond available work goes unused. These totals do not predict individual queue age, deadline attainment or review quality.

Static default calculation

20 cases/day × (75% × 10 + 25% × 30 minutes) = 300 minutes/day. Four reviewer-hours supply 240 minutes/day. Starting with zero, outstanding effort is 60, 120, 180, 240 and 300 minutes after days 1–5.

Case mix and handling time determine incoming effort. Unfinished work remains outstanding; increasing capacity in this arithmetic does not establish review quality or a staffing recommendation.

Universal sign-off can create an unmanageable queue instead of meaningful verification. Reduce avoidable exceptions through better inputs and clearer rules, reserve review for decisions that need it, and keep the actual review burden visible. Paperclip's liveness discussion frames this as a tension between continuing useful work and enforcing valid progression, not a reason to abandon either.

Manual takeover must address execution as well as assignment. Stop new automated commitments, determine which attempts remain in flight and reconcile their effects before conflicting manual work proceeds. Cancellation can be cooperative, and a timed-out attempt may continue. Marking a task human-owned therefore does not establish exclusive control over external records. The implementation must separately prevent conflicting writes at those boundaries.

Integration and recovery

Preserve meaning across applications

An integration contract states what information and operations cross an application boundary and what their results mean. It must cover record identity, field meanings, permitted reads and writes, freshness, operating limits and confirmation. A connector that transports data successfully can still select the wrong account, misinterpret a status or submit an unauthorized change. Preserving meaning across integrations explains the domain agreement; authorization at the protected operation explains enforcement.

Robotic process automation, or RPA, performs prescribed interactions with existing software, often through its graphical interface; it need not use AI. Mary Lacity, Leslie Willcocks and Andrew Craig's April 2015 Telefónica O2 case study describes a 2010 trial handling SIM swaps and precalculated credits by logging into applications and processing transactions. O2 considered RPA alongside process-management software and customer self-service, while retaining some work manually. Interface automation was an implementation option, not a replacement for process design.

Choose the interface from the workflow requirement. For an API, establish supported operations, identity mapping, permissions, limits and result semantics. For notifications, establish delivery and freshness behavior. For batch exchange, define the acceptable delay and per-record acceptance reporting. For graphical automation, identify the intended controls and how their results will be checked. These are requirements to verify for the actual integration, not guarantees implied by the interface category.

When several interfaces retrieve equivalent information, convert their outputs into a shared format before applying business rules. This separates how information is obtained from what decision it supports. In Risa Labs' oncology workflow, an API or automated portal interaction retrieves insurance coverage information. Both paths produce the same coverage-result format for a deterministic check of whether coverage is active; a failed check stops progression. The shared format lets both paths use the same rule, but cannot establish that the retrieved values are correct. Computer Use explains graphical execution and verification.

The architecture described by Risa Labs uses alternative API and portal access paths, then a shared format and active-coverage rule. Normalization makes one rule possible; it does not verify the retrieved values.

Confirmation must reach the business boundary. Under HTTP semantics, 202 Accepted means processing is incomplete, not that the requested change succeeded. Likewise, repairing a broken interface selector restores a way to attempt an action, not proof of its result. Microsoft's UI automation repair guidance describes operator review and testing of proposed selector repairs during debugging. Maintenance and business-result verification remain separate work.

Recover from the effects that occurred

Recovery starts from three different claims: confirmed success, confirmed failure without an effect, or an unknown outcome. A missing response cannot distinguish them. Reconciliation resolves the uncertainty or discrepancy against authoritative records. Preserve the logical operation identifier and determine what happened before selecting the next business action. Recover when the effect is unknown develops the interface mechanics.

Consider an attempted external record creation whose response times out. The record might exist, or the operation might not have committed. An authoritative lookup or operation-status check can justify continuing from an existing record; confirmed non-execution can justify considering another authorized attempt. An unavailable lookup or incomplete search leaves the outcome unknown. Stripe's low-level error guidance illustrates why even some server errors remain indeterminate and why local operation identifiers help correlate later provider evidence.

Idempotency means repeated attempts do not add another effect beyond the first. It needs enforcement by the receiving operation, not merely a logged key. Temporal's activity documentation identifies the effect/reporting gap: an external effect can occur before a worker dies without recording completion. A retry can then repeat it. Recorded completion protects replay; unrecorded completion still requires safe repetition or reconciliation.

Same local symptom, different external states

Same local symptom, different external statesTwo alternative histories of logical create O1 have no success recorded. In A the application commits, returns a result, then the worker dies before recording completion; the record remains. In B the request is lost before arrival and never commits. Both local outcomes are unknown until reconciliation.A · committed, completion unreportedWorkerReceiving applicationExecution historyRequest · create O1Record committedResult · record IDCompletion report lost · worker diesCommitted record remainsNo recorded successSame local observation: O1 outcome unknown.B · request never arrives, no commitWorkerReceiving applicationExecution historyRequest · O1 lostNo record createdAttempt times outNo recorded successSame local observation: O1 outcome unknown.

Reconcile the same O1 against authoritative receiving-system state. A timeout does not stop an in-flight operation, and an incomplete search cannot prove non-execution.

Required effect confirmed

Continue remaining work using the existing result.

Non-execution authoritatively confirmed

Consider a repeat only with valid authority and safe operation semantics.

Unavailable or inconclusive

Retain owned investigation; the outcome stays unknown.

These are alternative histories of the same logical operation O1. Losing a completion report does not undo an external commit. Reconciliation distinguishes confirmed effect, confirmed non-execution and unresolved investigation; recording O1 alone does not enforce idempotency.

Partial completion is still real

Compensation performs a new business-specific action to address completed work. It is not database rollback: concurrent changes must survive, the original state may be unattainable and cancellation can incur charges. Record completed effects and the information needed to compensate them. Compensation can itself fail, so preserve its progress and assign manual recovery when necessary. For irreversible effects, define an acceptable remedy rather than claiming erasure. Microsoft's Compensating Transaction pattern explains these boundaries.

If one application has committed and another has failed, recover those specific obligations rather than restarting the whole business process. A completed charge followed by failed receipt delivery calls for recovering the receipt step, not casually charging again. Eric Allam's replay example illustrates preserving completed steps, subject to the external-effect gap just described.

Durable execution preserves logical progress across interruptions, including long human waits. It can let a process resume without keeping one worker alive. It does not establish that external effects were reversed when execution was cancelled, or that an old approval is still valid. On resumption, recheck relevant business state and authorization; investigate late completions rather than discarding them. Persistence, replay and retry implementation belong in Agent Runtimes and Harness Engineering.

Outcomes and daily operation

Measure the complete process

Return to the case boundary and customer baseline. Measurement should describe useful completed work, including checking, waiting and correction. Klarity's outcome-first evaluation approach works backward from user value to utilization and feature-health checks; a valid output format is one useful check, not a substitute for the user's result.

Record these dimensions separately for the baseline and redesigned process. Use the same eligibility rules, work mix and observation window; leave unavailable observations explicit.
MeasureDefinition for the comparisonWhat it can hide alone
Elapsed completion timeTime from the defined start to accepted completion, including waitingStalled cases absent from a completed-only sample
Active handling effortPerson-time spent producing, checking, correcting and recovering workConcurrent effort if mistaken for elapsed time
ThroughputAccepted useful completions per unit of timeA changing case mix or weakened acceptance standard
First-pass yieldCases meeting requirements without repair divided by cases entering the defined processEventual quality after correction; unresolved cases need separate visibility
Straight-through coverageCases completed without human intervention divided by eligible entering casesWhether those results were correct
Unresolved workOutstanding cases, their ages, reasons and owners at the observation cutoffNothing should silently convert these cases into successes
Operating cost per accepted completionAttributed operating costs, including review, repair and integration maintenance, divided by accepted completionsUnfinished work and one-time delivery costs unless reported alongside

ASQ's first-pass-yield definition excludes work requiring reruns, retesting or repair. A case can therefore complete eventually without passing on its first attempt. Likewise, a fully automatic path can be wrong, while a human-assisted path can meet the requirement immediately. If two people work simultaneously for ten minutes, the process consumes twenty person-minutes but only ten elapsed minutes during that interval. Keep these quantities separate.

Interpret the comparison

Generative AI at Work, by Erik Brynjolfsson, Danielle Li and Lindsey Raymond, supplies a bounded example of useful assistance. Its November 2024 revision studies staggered deployment among 5,172 support workers and reports a 15% average increase in issues resolved per hour. Workers could edit or ignore suggested responses. Effects differed: less experienced and lower-skilled workers improved speed and quality, while the most experienced and highest-skilled workers had small speed gains and small quality declines. This is evidence about one firm's assisted workflow, not autonomous fulfillment or net operating savings.

Released capacity is not automatically reduced expenditure. If staff remain employed and use freed time for more work, capacity has increased without a payroll reduction. The Government Efficiency Framework distinguishes these benefits and requires realized savings to account for delivery costs and costs shifted elsewhere. Report the actual benefit: shorter waits, more completions, better quality, less effort or lower spending.

A credible comparison must also account for changes in case difficulty, staffing and policy. Faster completed cases do not establish improvement if hard cases accumulate unresolved. Choose an appropriate live experiment, and connect execution records to actual outcomes through Observability. Component speed is useful for diagnosis; the process boundary determines the benefit claim.

Keep the workflow usable and owned

Adoption means sustained incorporation into eligible work. An operational measure is the share of eligible cases actually handled through the intended arrangement over a stated period, accompanied by bypasses and duplicate manual work. Account creation and occasional use answer different questions. Normalization Process Theory emphasizes the everyday interactions, confidence, accountability, division of labor and resources needed to make a practice workable.

Observation-only operation, assistance and narrowly authorized automation are legitimate long-term choices. Shadow mode compares a candidate's behavior without adopting its outputs. SageMaker shadow testing, for example, copies inference requests to a candidate while returning production responses to the application. That response-routing contract does not make a shadow agent's tool calls harmless. During comparison, keep one authorized path for real effects and prevent the candidate from duplicating them.

Operators need practical training, a usable manual route and support that can resolve problems. If they routinely repeat the entire task to trust the output, include that effort in the result and investigate its cause. Preserving a recognizable process can help people understand their new responsibilities, but familiarity should not protect unnecessary work. Test the arrangement with its actual operators.

Assign responsibility for maintaining the process, not merely for launching its software.
Observed problemResponsible rolePossible change
Cases bypass the intended pathProcess owner with operatorsRevise eligibility, task fit or unnecessary handoffs
Rules no longer match the workDomain decision ownerReview policy and update its executable checks
Application changes break accessIntegration ownerRepair and verify the affected boundary
Exceptions age without actionOperations ownerAdjust staffing, priorities, escalation or admitted scope
Operators repeatedly correct the same outputQuality owner with domain specialistsIdentify whether the source, rule, interface or model behavior needs repair

Corrections need an explicit destination. Fixing the present artifact, reporting a defect and changing future model behavior are different operations. Google's Feedback + Control guidance recommends promising only the timing and scope of improvement the system can deliver. Human feedback does not automatically train a model. A recurring problem may instead require a source correction, clearer rule, better interface or separately evaluated model change.

Use operating handoff to establish that the receiving team can run and recover the process; use Enterprise AI for wider organizational adoption. Then revisit the original allocation using observed completion, quality, burden and maintenance. Retain automation where it helps, revise the arrangement where it moves the bottleneck, and stop it where the complete process is worse. Greater autonomy is one possible change, not the destination.

Open questions

  1. Sustained net benefit remains difficult to establish when task difficulty, review effort, staffing and adoption change together. Progress would mean longitudinal comparisons that retain unresolved cases and measure accepted completion, correction burden and full operating cost on a consistent eligible population.

  2. Manual takeover across multiple applications remains harder than assigning a human task. In-flight automation and independent writers can outlive that assignment. Progress would mean implementation-specific demonstrations that prevent conflicting commitments while reconciling attempts whose effects are still unknown.

  3. Review allocation must preserve error detection without producing a queue people cannot meaningfully inspect. The difficulty is that review policy changes both the selected workload and reviewer behavior. Progress would combine error-detection tests with queue age, handling effort and downstream outcomes, rather than optimizing approval rate alone.

  4. Process records reveal only the work that systems capture. Informal clarification, duplicate checking and off-system repair can make a redesigned path appear better than it is. Progress would combine event-based analysis with targeted observation and explicit accounting for omitted work.

Follow the curated reading path through the speakers and demonstrations behind this entry.

15 min

AI Engineer Summit 2025 · 2025

Missing pieces of workflow automation

Shirsha Chaudhuri

Cited in this entry

Use its support and content examples to recognize when task-by-task automation leaves the original handoffs and approval structure untouched. It motivates redesign with domain specialists.

Watch talk
18 min

AI Engineer World's Fair 2026 · 2026

Your Agent Didn’t Fail. Your Harness Did.

Vinoth Govindarajan

Cited in this entry

Connects workflow responsibility to state ownership, scoped approval and records of actual external effects. Useful when investigating a process that appears successful but cannot establish what happened.

Watch talk
28 min

AI Engineer World's Fair 2024 · 2024

E-Values: Evaluating the Values of AI

Sheila Gulati · Nischal Nadhamuni

Cited in this entry

Shows how to work backward from user outcomes to an evaluation collection, instead of treating convenient feature-health metrics as business value.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

45 matching talks

TalkSpeakerEventYear
Balázs HorváthAI Engineer World's Fair 20262026
Cornelia DavisAI Engineer World's Fair 20262026
Martin Harrysson, Natasha ManiarAI Engineer Code 20252025
Eliza Cabrera, Jeremy SilvaAI Engineer World's Fair 20252025
DottaAI Engineer World's Fair 20262026
Dan MasonAI Engineer World's Fair 20252025
Rachna SrivastavaAI Engineer World's Fair 20252025
Dan BjornnAI Engineer World's Fair 20262026
Steven WillmottAI Engineer Europe 20262026
Elizabeth Fuentes LeoneAI Engineer World's Fair 20262026
Anju KambadurAI Engineer Summit 20252025
Dan FarrellyAI Engineer World's Fair 20262026
Ben SteinAI Engineer World's Fair 20252025
Angel Ortmann LeeAI Engineer World's Fair 20262026
Jeremy Silva, Chris HernandezAI Engineer World's Fair 20252025
Keegan McCallumAI Engineer World's Fair 20252025
Anant ShankhdharAI Engineer World's Fair 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Eric AllamAI Engineer World's Fair 20252025
Justin ReockAI Engineer Code 20252025
Sachin GuptaAI Engineer World's Fair 20262026
Nikhil AbrahamAI Engineer World's Fair 20252025
Mike SpitzAI Engineer Europe 20262026
Anna Marie BenzonAI Engineer World's Fair 20262026
Alex VolkovAI Engineer World's Fair 20262026
Vasuman MozaAI Engineer World's Fair 20262026
Garrett GalowAI Engineer Europe 20262026
Gabe De MesaAI Engineer World's Fair 20262026
Luke AlvoeiroAI Engineer Europe 20262026
Notion's Token Town

Transcript reviewed

Sarah SachsAI Engineer World's Fair 20262026
Kyle MisteleAI Engineer World's Fair 20262026
Mason EggerAI Engineer World's Fair 20252025
Jia WuAI Engineer World's Fair 20262026
How to Kill the Code Review

Transcript reviewed

Ankit JainAI Engineer World's Fair 20262026
Maggie AppletonAI Engineer Europe 20262026
Content Is Code

Transcript reviewed

Matt PalmerAI Engineer World's Fair 20262026
Chaitanya AsawaAI Engineer World's Fair 20262026
Sarthak AggarwalAI Engineer World's Fair 20262026
Samuel ColvinAI Engineer Code 20252025
Cornelia DavisAI Engineer Code 20252025
Yohei NakajimaAI Engineer World's Fair 20262026
Apoorva JoshiAI Engineer World's Fair 20262026
Sarmad QadriAI Engineer World's Fair 20252025
Nicholas ArcolanoAI Engineer Code 20252025
Dan FengAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
50 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
0 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. ISO 9000: Quality Management Vocabulary

    ISO defines a process as interacting activities that use or transform inputs to deliver a result. Outputs from one process commonly become inputs to another. A process owner is the person or team responsible for defining and maintaining the process; responsibility can exist at different organizational and project levels. Its workflow entry describes the activities necessary to complete a task and distinguishes partially automated from fully automated execution.

  2. Missing pieces of workflow automation

    Evaluate the whole business process rather than merely inserting agents into its existing tasks.

  3. OMG: Case Management Model and Notation, Version 1.1

    CMMN distinguishes a case model from a particular instance, such as one invoice discrepancy. A case file contains or references structured data and documents used in decisions. New facts, received documents, task completion, and milestones can prompt decisions about subsequent work. Rules can constrain actions and state transitions using case data and current state. Human knowledge and explicit business rules both influence case handling.

  4. Zendesk: What is the difference between a solved ticket and a closed ticket?

    Agents can mark a ticket solved after resolving the issue or completing work. Closure is performed through automation or triggers. A reply can reopen a solved ticket, but a reply to a closed ticket creates a follow-up while the original remains closed. Thus, for this documented workflow, a closed status records a workflow transition and does not by itself establish customer-confirmed resolution.

  5. Missing pieces of workflow automation

    AI practitioners should redesign workflows together with domain specialists.

  6. Toyota Production System

    Toyota describes two complementary principles: jidoka stops work when an abnormality is detected, while Just-in-Time coordinates stages to produce what is needed, when needed, in the needed quantity. Its account traces automatic stopping to Sakichi Toyoda’s 1896 power loom and describes Kiichiro Toyoda introducing flow production at the Koromo plant in 1938. Taiichi Ohno subsequently helped establish the production system with support from Eiji Toyoda. Toyota’s present explanation recommends understanding and improving manual work before mechanizing it. Automatic abnormality detection reduces the need for continuous machine watching; synchronized flow addresses waiting and excess inventory across stages.

  7. The New Industrial Engineering: Information Technology and Business Process Redesign

    Thomas H. Davenport and James E. Short’s June 1990 report, also published in Sloan Management Review that summer, argues that information technology and process redesign should inform each other rather than merely accelerate isolated tasks. Drawing on research in nineteen companies, it defines processes around outcomes and recipients across organizational boundaries. Its order example distinguishes creating an order from the more useful outcome of receiving goods. The authors propose setting business objectives, selecting processes, understanding and measuring current work, identifying technological possibilities, and designing and prototyping the new process. They describe organizations whose departments optimized their own systems while nobody owned the complete process.

  8. Workflow Management Coalition: The Workflow Reference Model

    David Hollingsworth’s archived Workflow Reference Model identifies itself as Issue 1.1 dated 29 November 1994. It addresses incompatible workflow products through common terminology, components, and interfaces. Workflow systems coordinate documents, information, and tasks among human and software participants under procedural rules. The model separates process definition at build time, sequencing and control during execution, and interactions with people and applications. It explicitly distinguishes redesign from automation: redesign need not produce a workflow implementation, and a workflow system can automate an unchanged procedure. Separating procedure logic from operational support permits subsequent changes to the procedure.

  9. A Model for Types and Levels of Human Interaction with Automation

    Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens’s May 2000 paper separates automation into information acquisition, information analysis, decision and action selection, and action implementation. Each function can receive a different degree of automation. A system can therefore automate collecting and organizing information while leaving consequential choices or execution with a person. The proposed framework evaluates allocations through human-performance consequences, automation reliability, and the costs of incorrect actions. It considers workload, situation awareness, trust, and skill rather than treating greater automation as an unconditional improvement.

  10. Lean Enterprise Institute: Value Stream Mapping

    Value-stream mapping follows material and information from order to delivery, including activities that do not create customer value. Teams first capture actual current flow and then specify desired future flow. The institute emphasizes improving the whole flow rather than optimizing isolated steps. Lead time measures how long one item takes to traverse the defined process; cycle time describes an operator completing work elements before repeating them. Process data also include availability and the proportion completed accurately.

  11. AHRQ: Mapping and Redesigning Workflow

    AHRQ distinguishes perceived, observed, and desired processes. Participants describe their own steps, then direct observation checks what actually happens, including variations. Detailed flowcharts include decisions, waiting, and feedback; swimlanes organize work across roles and stages. Redesign questions examine unnecessary steps, duplication, frequency, required skills, breakdowns, and alternative ways to accomplish the task. Changes can affect dependent processes and require revised staffing, time allocation, responsibilities, training, and accountability. The guide recommends testing changes before expanding them.

  12. Leadership in AI-Assisted Engineering

    Apply Eli Goldratt's Theory of Constraints to find the workflow bottleneck; the speaker gives legacy-code reverse engineering as a concrete target.

  13. One Developer, Two Dozen Agents, Zero Alignment: Why we Need Collaborative AI Engineering

    Move team alignment before agent execution and maintain it throughout implementation rather than making the pull request the first substantive discussion.

  14. Process Mining Manifesto

    Wil van der Aalst and fellow IEEE Task Force contributors describe process mining as learning about actual processes from recorded events. Discovery constructs a process model; conformance checking compares recorded behavior with a model; enhancement improves an existing model using event information. Events identify activities and process instances, with timestamps and resources enabling additional analyses. The case boundary depends on the question: an order, order line, and delivery can produce different views because their relationships are not necessarily one-to-one. The manifesto requires trustworthy, sufficiently complete logs and warns that missing or poorly recorded activities limit conclusions.

  15. Building Applications with AI Agents

    Keep state and fixed business rules outside the model, and let the agent update validated state through tools.

  16. AI tools for Forward Deployed Engineering

    Redesign individual steps for autonomous, human-assisted, or fully human execution while preserving enough familiarity for operators to adopt the system.

  17. Building Applications with AI Agents

    Start with chains or branching workflows; consider fuller agency when explicit orchestration becomes too convoluted to maintain.

  18. Robotic Process Automation at Telefónica O2

    Mary Lacity, Leslie Willcocks, and Andrew Craig’s April 2015 case study describes Telefónica O2’s use of robotic process automation: software performs prescribed interactions with existing applications through their user interfaces. A 2010 trial covered SIM swaps that retained customers’ telephone numbers and applying a precalculated credit. The software logged into applications, processed the transaction, and logged out. O2 considered RPA alongside business-process management software and customer self-service, not as their universal replacement. Its approach included eliminating or simplifying work before automation and retaining smaller workloads as manual work when local economics did not justify automation.

  19. The Build-Operate Divide: Bridging Product Vision and AI Operational Reality

    Prioritize human-in-the-loop review at consequential decision points in high-risk, high-trust workflows.

  20. Decision Model and Notation, Version 1.3

    A decision table represents rules connecting input conditions to outputs. Its hit policy defines what happens when several rules match. DMN’s default Unique policy prohibits overlapping rules; Any permits overlap only when matching rules produce equal outputs. First returns the first matching rule, making row order significant. The specification warns that First tables can conceal inconsistencies and make manual validation harder. This provides a concrete representation for explicit business decisions whose conditions and outcomes can be stated, separately from the surrounding sequence of work.

  21. Anthropic: Building effective agents

    Anthropic distinguishes workflows that coordinate models and tools through predefined code paths from agents in which a model directs subsequent processing and tool use. Its guidance recommends starting with the simplest sufficient solution, including avoiding agentic systems when unnecessary. Added autonomy can trade greater latency and cost for task performance; predefined workflows suit well-specified tasks.

  22. What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip

    Use explicit task transitions and enforced dependencies, with invariants that preserve productive progress, stop only for real blockers, and bound loops.

  23. IT Admin for the AI Workforce — Sarthak Aggarwal, Decawork

    Separate planning from execution so that reasoning over content does not itself confer authority to invent and execute actions.

  24. Challenges to Scaling Agents for Generative AI Products

    Bloomberg's semi-agentic architecture makes policy and factuality guardrails mandatory rather than allowing agents to decide whether to invoke them.

  25. IBM Redbooks: Building the Operational Data Store on DB2 UDB

    IBM’s operational-data-store architecture makes system-of-record responsibility a business decision for particular data elements, such as a customer address. When updates travel between systems, explicit rules determine which incoming values may overwrite the designated record. Different source systems may retain authority over different elements. Bidirectional synchronization can create cyclical updates. The guide describes attaching an identifier for the updating application or person and using that audit information to recognize and bypass already synchronized records. Its integration design also separates change capture from applying changes and requires handling unavailable destinations and data integrity.

  26. Your Agent Didn’t Fail. Your Harness Did.

    The harness should own state transitions, authority checks, ordered commits, and durable evidence; the model proposes actions.

  27. Your Agent Didn’t Fail. Your Harness Did.

    Delivery alone is insufficient: a named system of record must persist the fact and support replay into future work.

  28. Your Agent Didn’t Fail. Your Harness Did.

    Trace one real run from trigger identity through inherited state, authority, execution attempts, and surviving external evidence.

  29. From Stateless Nightmares to Durable Agents

    Logfire instrumentation exposed nested agent calls and the workflow ID, allowing the demo to attach to the interrupted workflow instead of launching another one.

  30. RFC 9110: HTTP Semantics

    HTTP 202 means a request was accepted for processing but processing is incomplete; it may subsequently be disallowed. The response ought to describe status and identify or include a status monitor. Conditional requests can prevent one client from overwriting another client’s concurrent changes. If-Match compares supplied entity tags with the current representation using strong comparison, making the operation conditional on the represented state.

  31. Your Agent Didn’t Fail. Your Harness Did.

    Provide one ordered commit path per mutable state boundary while allowing independent work to run concurrently.

  32. Exposing Agents as MCP Servers with mcp-agent: Sarmad Qadri

    Agents that pause for humans, require retries, or start from scheduled events should be modeled as asynchronous workflows; mcp-agent uses Temporal for durable execution.

  33. W3C: State Chart XML—State Machine Notation for Control Abstraction

    SCXML separates an event that triggers consideration of a transition from its guard, a Boolean condition controlling whether the transition is permitted, and its target state. Parallel regions can remain active simultaneously. Completion of one region does not complete the enclosing parallel state: its completion event occurs when all child regions have reached final states.

  34. Stripe: Receive events in your webhook endpoint

    Stripe does not guarantee event delivery order and may deliver an event repeatedly. It recommends tracking processed event IDs; distinct event objects can sometimes represent duplicates identified through object identity and event type. Snapshot-event timestamps have second-level precision, so timestamps alone cannot establish ordering or deduplication. Existing event objects do not change when their underlying resources change. Missing resource information can be retrieved through the API.

  35. Web Services—Human Task Specification Version 1.1, Committee Specification 01

    OASIS’s 17 August 2010 WS-HumanTask specification models human work as tasks with inputs, assignments, lifecycle states, and results. A task available to several potential owners or a work queue enters Ready; claiming it assigns an actual owner and changes it to Reserved. Starting work changes it to InProgress, separately from completion or failure. If assignment yields nobody, an administrator must nominate potential owners before normal progress. Start and completion deadlines can trigger escalation. The published approval example sends reminders, informs a manager, or reassigns work when a start deadline is missed.

  36. PagerDuty: Incidents

    PagerDuty separates triggering, acknowledging, and resolving an incident. Assignment and notification follow an escalation policy; acknowledgment records that a responder claims ownership and is working on the unresolved issue. Without acknowledgment, escalation continues. An acknowledgment timeout can return the incident to triggered status and resume escalation. Incident timelines record status changes, actions, and notifications. This supplies an operational example in which requesting attention, accepting responsibility, and resolving work are distinct events.

  37. Business Process Model and Notation, Version 2.0.2

    BPMN provides distinct graphical elements for different workflow relationships. Activities represent work; events represent occurrences affecting flow; gateways control branching and joining. Sequence flows specify activity order within a process. Message flows represent communication between participants, shown as separate pools. Lanes organize and categorize activities within a process. These distinctions let a map show who participates, what work occurs, and where communication crosses participant boundaries without representing every connection as the same kind of arrow. The specification also defines process interchange and execution semantics.

  38. What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip

    Make blockers, reviewers, approvers, and audited human choices first-class parts of task progression.

  39. AI System Design: From Idea to Production

    Expose the recommendation's supporting citations and collect both verdict overrides and citation-specific feedback.

  40. The Build-Operate Divide: Bridging Product Vision and AI Operational Reality

    Automated evaluations do not eliminate the human review capacity bottleneck identified by the speakers.

  41. Building Your Own Secure AI Workflows: Human-in-the-Loop Automation with n8n

    Place approval in the execution path so the agent cannot bypass it by choosing to call the underlying tool directly.

  42. OWASP Transaction Authorization Cheat Sheet

    Transaction authorization should let the user acknowledge significant transaction data, such as destination and amount, and bind approval to that transaction rather than an unrestricted session. The server controls authorization data and permitted state transitions. Changes to transaction data invalidate prior authorization or restart the process. Credentials should be unique per operation and valid only for a limited interval. A final server-side gate immediately tied to execution verifies that the transaction was properly authorized, preventing skipped checks and substitution between approval and use. For an agent, approving a draft action therefore must not silently authorize a changed target, payload, or scope.

  43. When Machines Mislead: Human Review of Erroneous AI Cheating Signals

    The researchers introduced fabricated copy-typing alerts into previously certified test sessions and examined proctor decisions in two studies. Revised instructions explicitly required independent suspicious behavior before upholding an alert. Model-estimated rejection of fabricated alerts increased from 50% to 71%; rejection of genuine operational alerts also increased. The study demonstrates a way to investigate whether reviewers challenge erroneous assistance instead of measuring approval alone.

  44. UiPath Orchestrator: Business Exception Vs Application Exception

    UiPath distinguishes technical application failures from business exceptions caused by missing or invalid input. Its example contrasts a frozen receiving application with an incomplete telephone number. Retrying may resolve the former; repeating the unchanged incomplete number cannot repair it. Application exceptions can trigger configured automatic retries, whereas business exceptions are not retried by default and can require human action.

  45. AI tools for Forward Deployed Engineering

    Interview process leads about exception handling and actual handoffs before designing automation.

  46. Temporal Activity Execution

    An Activity Execution can comprise multiple task attempts. Temporal relies on timeouts to detect lost work, including worker crashes after invocation, and retries according to policy; limiting attempts to one prevents retry but does not prove an external effect failed. Cancellation is cooperative: activities receive service cancellation through heartbeats, can ignore it, and workflows may proceed without waiting for acceptance. A timed-out attempt may therefore continue while another attempt runs. Application consequence: treat an unconfirmed external mutation as uncertain, retain its operation identifier, reconcile against the receiving system, and use enforced idempotency or explicit recovery before repeating it. Timeout or cancellation is not evidence that a payment, message, or write was reversed.

  47. UiPath Orchestrator: Review Requests

    Orchestrator provides a review worklist containing queue items assigned to the logged-in reviewer. Revision states distinguish items under review from items marked verified or selected for retry. Marking an item verified prevents further retry through this mechanism. Removing an item from the review display marks it deleted in transaction records without deleting its database record.

  48. Amazon Builders’ Library: Avoiding insurmountable queue backlogs

    Amazon describes how an upstream producer can accumulate work faster than downstream consumers can process it, leaving long recovery periods after a short surge. Its techniques include workload isolation, admission controls, prioritization, and backpressure that reduces accepted upstream work when consumers fall behind. Backpressure is not universally appropriate: the report describes order-processing systems that prefer accepting orders while prioritizing urgent work.

  49. What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip

    Exhaustive human review can become a queue bottleneck and verification theater when agents produce work faster than people can inspect it.

  50. Your Agent Didn’t Fail. Your Harness Did.

    Approval must remain bound to one specific action and its scope, identity, arguments, and lifetime; expiration should terminate the approval path.

  51. Microsoft Power Automate: AI-assisted UI automation repair

    Microsoft documents maintenance needs when application interfaces change or selectors stop uniquely identifying intended elements. Its repair workflow locates the failed action, asks the operator to recapture the intended element, and proposes an updated selector. The operator can review, test, modify, accept, or skip that proposal. The documented feature operates during debugging and does not automatically repair attended or unattended runtime execution.

  52. Can Oncology Workflows Run Without Human Touch? - Anant Shankhdhar, Risa Labs

    Route retrieval through an API or RPA path, normalize both outputs, and apply a shared deterministic eligibility gate.

  53. Resolving an ambiguous payment request

    A timeout can leave the client unable to tell whether Stripe received or executed a request. Stripe documents retrying with the same key and parameters until a server result is obtained, using backoff. An HTTP 500 remains indeterminate: side effects may exist even though the cached response stays unchanged. Stripe may reconcile partial mutations and emit webhook events for resulting objects. Supplying a local operation identifier in metadata lets the application correlate these objects with its own pending operation. Engineering consequence: preserve pending state until authoritative provider evidence resolves it; do not infer failure solely from a timeout.

  54. Temporal Activity Definition

    An activity can complete an external effect and then lose its worker before reporting completion. Temporal's history then lacks successful completion, so a retry may execute the activity again. Completed activities recorded in history are not re-executed by workflow replay; unrecorded completion is the important gap. Idempotency means repeated attempts leave no additional effect beyond the first. Temporal documents idempotency keys enforced by the called service, rather than automatically by the activity. Retrying a multi-step activity repeats earlier successful steps too.

  55. Compensating Transaction pattern

    Compensation performs new, business-specific actions to counter completed steps of an eventually consistent workflow. It differs from transaction rollback: intervening concurrent work must be preserved, the exact original state may be unattainable, and cancellation may incur charges. Record completed steps and the information needed to compensate them. Compensation order need not exactly reverse execution, and some steps can run in parallel. Compensation can itself fail, so persist progress, resume from failure, and make retryable steps idempotent. Where automated recovery is impossible, alert an operator with diagnostic information. For irreversible effects, an application must define an acceptable remedy or escalation rather than claim the action has been undone.

  56. Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

    The presented replay model caches completed side-effect steps so a retry can skip them and reach the failed operation.

  57. Building Durable, Production-Ready Agents with OpenAI SDK and Temporal

    Temporal lets a workflow represent a logical ongoing process while worker memory can be released during a long wait.

  58. E-Values: Evaluating the Values of AI

    Design evaluations backward from the user outcome rather than assuming an easy technical metric represents the experience.

  59. Brynjolfsson, Li, and Raymond: Generative AI at Work

    The inspected November 2024 revision studies staggered deployment of assistance among 5,172 customer-support workers. The assistant suggests responses, while workers remain responsible and can edit or ignore suggestions. The authors report a 15% average increase in issues resolved per hour, separating handling time, chats handled per hour, and resolution share. Effects differ across workers: less experienced and lower-skilled workers improve speed and quality, while the most experienced and highest-skilled workers show small speed gains and small quality declines.

  60. ASQ: Quality Glossary—First pass yield

    ASQ defines first-pass yield as the proportion of units completing a process to its quality requirements without scrap, reruns, retesting, returns, or diversion into offline repair. Its denominator is the units entering the process. This distinguishes successful completion on the first pass from an eventual success count that conceals repair work.

  61. The Government Efficiency Framework

    The UK Government Efficiency Framework distinguishes reduced spending from benefits that increase capacity without reducing spending. Its published example keeps ten staff while a new system raises processing from 100 to 150 customer queries per hour: this is a monetisable non-cash-releasing benefit, not a budget reduction. Reported cash savings must be realized, calculated net of delivery costs, avoid double counting and unintended costs elsewhere, and preserve performance and outcomes. Merely moving costs into a later year does not establish an efficiency gain.

  62. What Data from 20 Million Pull Requests Reveal About AI Transformation

    Track changes within a team under stable conditions rather than comparing absolute PR counts across architectures.

  63. What Data from 20 Million Pull Requests Reveal About AI Transformation

    Company-week observations associated greater AI adoption with higher per-engineer PR throughput and shorter PR cycle times.

  64. Normalization Process Theory: Collective Action

    Normalization Process Theory describes implementation as operational work involving everyday interactions, confidence and accountability, division of labor, and allocation of resources. Its examples show how uncertainty about a tool can prompt duplicate checking and how assigning a task to people without relevant expertise can undermine its use. Resource allocation and organizational procedures are part of implementing a practice.

  65. Amazon SageMaker AI: Shadow tests

    SageMaker shadow testing routes copies of live inference requests to a candidate variant while returning only the production variant’s responses to the calling application. Candidate responses can be logged for comparison or discarded. The documentation identifies latency, error rates, and configuration behavior as reasons to test before promotion.

  66. The Build-Operate Divide: Bridging Product Vision and AI Operational Reality

    Treat the prototype-to-reliability transition as a recurring operational iteration loop, described here as crossing a quality chasm.

  67. People + AI Guidebook: Feedback + Control

    User feedback can inform team review or feed an improvement mechanism, but feedback controls do not necessarily update a model. A show-more-or-less control might only filter current content. The guide distinguishes acknowledgments by their promised timing and scope: receipt, future improvement for everyone, future personalization, or an immediate visible update. Products should promise only changes and timelines they can deliver and allow users to revise prior selections or reset personalization.

  68. Case Study + Deep Dive: Telemedicine Support Agents with LangGraph/MCP

    Feedback resumes the existing agent thread, lets the agent revise its proposal, and preserves the review requirement rather than increasing confidence to permit automatic execution.

  69. NIST AI RMF Core

    NIST connects measurement to deciding whether a system achieves its intended purpose and whether development or deployment should proceed. Risk responses include mitigation, avoidance and acceptance; remaining risks should be documented. The framework includes considering viable non-AI alternatives and assigning responsibility for superseding, disengaging or deactivating systems whose outcomes conflict with intended use. It also calls for evaluating measurement processes themselves and maintaining post-deployment monitoring, feedback, incident response, recovery and change management.