Long-Horizon LLM Agents — WebGPT, WebAgent, V-IRL, Operator, Computer Use: A Multi-Paper Review

Figure 3 of A Real-World WebAgent (arXiv:2307.12856), reproduced for editorial coverage.

1. Paper identity and scope

This is a multi-paper review tracing the architectural and methodological evolution of long-horizon LLM agents that interact with web browsers and computer interfaces. Five primary artefacts are covered:

1. WebGPT — Nakano et al., OpenAI, December 2021. arXiv:2112.09332 ¹ . Text-based browser, behaviour cloning plus rejection sampling against a learned reward model. The lineage’s starting point.
2. A Real-World WebAgent — Gur, Furuta, Huang, Safdari, Matsuo, Eck, Faust, Google DeepMind + University of Tokyo, July 2023, ICLR 2024 Oral. arXiv:2307.12856 ³ . Decomposed planning, HTML summarisation, and Python program synthesis with HTML-T5 plus Flan-U-PaLM.
3. V-IRL — Yang, Ding, Brown, Qi, Xie, NYU + HKU, February 2024. arXiv:2402.03310 ⁵ . Geolocation-grounded agent platform built on Google Maps Platform; nine exemplar agents and three benchmarks.
4. Anthropic Computer Use — Claude 3.5 Sonnet Model Card Addendum and launch communication, October 2024 ^{7 8} . Screenshot + mouse + keyboard action loop on a Linux desktop; 14.9% OSWorld single-screenshot, 22.0% with multi-step.
5. OpenAI Operator / Computer-Using Agent (CUA) — research preview and Operator System Card, January 2025 ^{9 10} . Screenshot-driven GPT-4o-based agent; 38.1% OSWorld, 58.1% WebArena, 87% WebVoyager.

Retrieval. Every paper’s arXiv abstract page, ar5iv HTML render where available, and the vendor announcement pages for Computer Use and CUA were fetched at writer-time. Anthropic’s news page resolved on a direct WebFetch; OpenAI’s system card and CUA research-preview pages returned HTTP 403 on direct WebFetch and were sourced through a structured WebSearch summary plus the publication’s earlier cached coverage of these launches. Where a benchmark number is only available through the vendor announcement (CUA’s 38.1% / 58.1% / 87%, Computer Use’s 14.9% / 22.0%), the article binds the number to both the vendor URL and the relevant benchmark paper for cross-reference.

Classification. Architecture proposal · Training method · Inference method · RL · LLM-based · Application · AI safety. The five papers span four architectural eras (text-browser, HTML-aware modular, geolocation-grounded multi-agent, screen-pixel computer-use) of the same broad family: long-horizon LLM agents that close real-world task loops through tool calls.

Technical abstract (in the publication’s voice). The lineage of long-horizon LLM agents that operate computers and browsers has compressed dramatically across the four-year window 2021–2025. WebGPT in 2021 wired GPT-3 to a text-only browser with a six-action interface and a single domain (long-form question answering), training the model with behaviour cloning and rejection sampling against a learned reward model; the best 175B variant was preferred 56% of the time over human demonstrators on ELI5 ¹ . WebAgent in 2023 broke the problem into three sub-modules — instruction decomposition (HTML-T5 finetuned with a mixture-of-long-span-denoising objective), HTML summarisation (HTML-T5 again, predicting data-ref tags), and Python program synthesis (Flan-U-PaLM emitting Selenium scripts) — and hit 65–80% completion across three real-world websites and 71.5% step success on Mind2Web ³ . V-IRL in 2024 grounded agents in geolocation-aware perception via Google Maps Platform, exposing where vision encoders break under cultural and geographic distribution shift ⁵ . Anthropic’s Computer Use in October 2024 abandoned HTML as the interface entirely, returning to the screen as the unified action surface and scoring 14.9% to 22.0% on OSWorld ^{7 8} . OpenAI’s CUA in January 2025 closed much of the gap to human performance on web tasks, scoring 87% on WebVoyager against a 100% human ceiling ⁹ . The arc bends towards generality (modality of interface), but the central problem — long-horizon reliability — remains open.

Primary research question. What architectural and training choices enable LLM-driven agents to reliably close long-horizon tasks that span many tool calls, multiple subsystems, and an evolving real-world environment? Each paper in the lineage proposes a different answer.

Core technical claim (per paper).

• WebGPT: behaviour cloning plus rejection sampling against a learned reward model produces a question-answering agent whose browsed answers are preferred to human demonstrators ¹ .
• WebAgent: decomposing a long-horizon web task into planning, HTML summarisation, and program synthesis, with a specialised long-context HTML encoder, beats monolithic prompting on real-world websites by over 50 percentage points ³ .
• V-IRL: grounding agents in geolocation-aware real-world data via Google Maps Platform surfaces failure modes — geographic and linguistic bias in vision encoders — that purely synthetic environments hide ⁵ .
• Computer Use: the screen is a sufficient interface; an LLM trained to interpret screenshots and emit cursor coordinates can drive an arbitrary computer through its existing GUI ⁷ .
• CUA / Operator: combining GPT-4o vision with reinforcement-learning-trained reasoning yields a single general action space that matches human-level success on WebVoyager (87%) and substantially advances OSWorld and WebArena ⁹ .

Core technical domains.

Domain	Depth in this review
LLM-as-agent paradigm	Deep
Behaviour cloning + reward modelling + PPO	Moderate
HTML / DOM representation for LLMs	Deep
Long-context encoder design (local + global attention)	Moderate
Mixture-of-denoisers pretraining	Moderate
Screenshot-driven GUI agents	Moderate
Geolocation-grounded multimodal agents	Moderate
Benchmark families: MiniWoB++, Mind2Web, WebArena, WebVoyager, OSWorld	Moderate
Prompt-injection defences for computer-use agents	Surface

Reader prerequisites. High-school algebra; familiarity with the LLM-as-agent paradigm (a model that calls tools in a loop) is helpful but not required because the Glossary in Section 2.5 covers every term. No prior knowledge of HTML or browser automation assumed.

How this review marks its registers. This article mixes four distinct registers, and each is labelled inline so readers can calibrate trust at every claim:

• “From the paper:” — directly traceable to the cited paper’s text, equation, table, or figure, with a <FootnoteRef />.
• [External comparison] — common-knowledge background or third-party verified facts independent of the paper under discussion.
• [Analysis] / [Reconstructed] — the publication’s own reasoned assessment or faithful reconstruction from partial disclosure.
• [Reviewer Perspective] — critical or speculative assessment that goes beyond what the paper proves.

The Glossary in Section 2.5 lists these labels with first-appearance pointers.

2. TL;DR and executive overview

TL;DR. Between 2021 and 2025 the LLM-agent stack moved from a text-only browser with six commands (WebGPT) to a general-purpose computer-using agent that matches humans on live web tasks (OpenAI Operator at 87% on WebVoyager). The middle of the arc — WebAgent’s three-module decomposition with a specialised HTML encoder, and V-IRL’s geolocation-grounded multimodal grounding — explains why progress happened: the interface narrowed from natural-language clicking to direct screenshot-and-cursor control, and the action space converged on a single general API while reasoning got stronger underneath. The benchmark numbers move from 56% human-preference on ELI5 (WebGPT, 2021) to 38.1% on OSWorld and 87% on WebVoyager (CUA, 2025), but the long-horizon reliability problem — multi-step tasks that span subsystems — is still open.

Executive summary. Long-horizon LLM agents must do two things well: perceive an evolving environment, and act in it across many sequential steps without losing the plot. Across five papers and four years, each lineage entry made one specific bet. WebGPT bet on text-browsing with PPO-tuned reward modelling; that worked for question-answering, not for arbitrary tasks. WebAgent bet on a domain-expert HTML encoder plus program synthesis; that beat monolithic prompting by 50+ percentage points on real websites but required a 540B program-synthesis model and bespoke HTML-T5 pretraining ³ . V-IRL bet on geolocation grounding via Google Maps; it surfaced the cultural and linguistic biases of vision encoders that synthetic environments hide ⁵ . Anthropic bet on the screen as a universal interface; the initial Computer Use scored 14.9–22.0% on OSWorld ^{7 8} . OpenAI’s CUA combined that universal interface with RL-trained reasoning and reached 38.1% OSWorld and 87% WebVoyager ⁹ . The publication’s reading: the durable contributions are WebAgent’s three-module decomposition and the screen-as-interface convergence. The open problem is long-horizon reliability, especially on workflows that span subsystems.

Five practitioner-relevant takeaways.

1. The screen has won as the agent’s primary perception modality. Computer Use (Anthropic, October 2024) and CUA (OpenAI, January 2025) both abandon HTML-DOM as the interface; the model looks at pixels and emits coordinates. Engineering teams designing agent infrastructure should default to screenshot-plus-action loops, with HTML/DOM as an optional structured side-channel rather than the primary interface ^{7 9} .
2. Modular decomposition still helps. WebAgent’s three-module split (plan, summarise, synthesise) beat single-LLM prompting by over 50 percentage points on real websites ³ . Even in the screen-pixel era, a planner-executor split is the architectural default for any non-trivial workflow agent.
3. The HTML-T5 mixture-of-long-span-denoising recipe is portable. Span lengths $\mu \in \{8, 64\}$ outperform standard T5 spans $\mu = 3$ on HTML inputs because short spans only mask syntactic tokens like </, id=, >; longer spans mask semantically meaningful HTML chunks like <form class=" ³ . The recipe transfers to any structured-text domain (XML, source code) where syntax tokens shouldn’t dominate the masking signal.
4. Benchmark families partition by interface. MiniWoB++ and Mind2Web score HTML-aware agents; WebArena and WebVoyager score browser-using agents on live web; OSWorld scores full computer use across Ubuntu/Windows/macOS ^{11 12 13 14} . Cross-benchmark comparison should respect the interface partition; a strong WebVoyager number does not imply a strong OSWorld number.
5. The 87% WebVoyager score versus 38.1% OSWorld score (same CUA model) shows that web-only tasks are nearly saturated while full computer-use tasks remain substantially below human performance ⁹ . Teams deploying agent products in 2026 should treat web-only workflows as the near-term mature surface and full-desktop automation as the medium-term frontier.

Pipeline overview. Across all five papers the inference-time pipeline is variations on the same agent loop: (a) observe — read browser text (WebGPT), HTML snippet (WebAgent), street view image + map (V-IRL), or screenshot (Computer Use, CUA); (b) reason — emit a sub-instruction or directly an action; (c) act — call a tool, click a coordinate, or run a Python script; (d) repeat. The five papers vary in what is observed, what action vocabulary is used, and how the model is trained. Training-time pipelines differ: WebGPT used BC + RM + PPO + rejection sampling; WebAgent used self-experience supervision plus domain-specific pretraining; V-IRL evaluates off-the-shelf models without retraining; Computer Use and CUA use undisclosed proprietary RL recipes.

2.5. Glossary

A short dictionary of every technical term used in the rest of the article. A curious reader without prior agent background should be able to navigate the rest of this article using only this table.

Term	Plain-English explanation	First appears in
LLM agent	A language model wired up to call external tools (browser, file-system, APIs) in a loop, chaining tool calls to complete a multi-step task.	Section 1
Long horizon	A task that requires many sequential steps (often dozens) without losing track of the overall goal.	Section 1
Behaviour cloning (BC)	Training a model to imitate human demonstrations of a task by supervised learning on (state, action) pairs.	Section 1
Reward model (RM)	A separate neural network trained to predict which of two outputs a human would prefer; used as a stand-in for direct human feedback during RL.	Section 1
PPO	Proximal Policy Optimisation; a reinforcement-learning algorithm that updates a policy while constraining how far it drifts from a reference.	Section 6
Rejection sampling	Generate many candidate outputs, then keep only the ones a reward model ranks highly; cheaper than RL for some settings.	Section 6
HTML-DOM	The structured tree representation of an HTML page; what a browser parses before rendering pixels.	Section 5
Span denoising	A pretraining objective where contiguous chunks (“spans”) of text are masked and the model must predict them; the original T5 recipe.	Section 6
Mixture of denoisers	A pretraining objective that mixes several span-length distributions (e.g., short and long spans), forcing the model to learn at multiple granularities.	Section 6
Local + global attention	An attention pattern where each token attends only to nearby neighbours (local) plus a small set of summary tokens that see the whole sequence (global), reducing cost from $O(n^2)$ to roughly $O(n)$.	Section 6
Program synthesis	The agent emits executable code (e.g., a Python Selenium script) rather than directly clicking; the code is then run by an external interpreter.	Section 5
Self-experience supervision	The agent generates its own training data by executing tasks, then is finetuned on the successful episodes.	Section 5
Screenshot	An image capture of what’s currently on screen; the input modality for screen-pixel agents like Computer Use and CUA.	Section 5
OSWorld	A benchmark scoring agents on full computer-use tasks across Ubuntu, Windows, and macOS.	Section 4
WebArena / WebVoyager	Web-only agent benchmarks; WebArena uses self-hosted sites (offline), WebVoyager uses live sites like Amazon and GitHub.	Section 4
Mind2Web	An offline benchmark of action prediction on 137 real websites across 31 domains, scoring step accuracy.	Section 4
MiniWoB++	An older benchmark of 56 small simulated web tasks; the most-used pre-2023 web-agent benchmark.	Section 4
Prompt injection	An attack where text on a web page tries to override the agent’s instructions; a major safety concern for computer-use agents.	Section 12
”From the paper:” prefix	Marks a sentence whose factual content is directly supported by the paper’s text, equations, tables, or figures.	Throughout
[Analysis] label	Marks the publication’s own reasoned assessment, distinct from what the paper itself claims.	Sections 11–13
[Reviewer Perspective] label	Marks a critical or speculative assessment that goes beyond what the paper proves.	Sections 11–12
[Reconstructed] label	Marks content the publication faithfully reconstructed because the paper only partially disclosed it.	Sections 6–7
[External comparison] label	Marks a comparison to prior work or general knowledge outside the paper itself.	Section 4 and 11

3. Problem formalisation

Notation table.

Symbol	Type	Meaning	First appears in
$\mathcal{O}_t$	Observation	The agent’s observation at step $t$ (text, HTML, screenshot)	Section 3
$\mathcal{A}_t$	Action	The agent’s action at step $t$ (browser command, sub-instruction, click, code)	Section 3
$\pi_\theta$	Policy	The agent’s policy, parameterised by $\theta$	Section 6
$T$	Horizon	Number of agent steps per task	Section 3
$r$	Reward	Scalar reward returned by the reward model (WebGPT) or env (RL papers)	Section 6
$\mathcal{D}_{\text{demo}}$	Dataset	Human demonstration trajectories used for BC	Section 6
$\mathcal{D}_{\text{pref}}$	Dataset	Pairwise human-preference data used to train the reward model	Section 6
$\mu$	Span length	Mean masked-span length in denoising pretraining	Section 6

Formal problem statement. Each paper instantiates a partially-observable sequential decision problem. The agent observes $\mathcal{O}_t$ at step $t$ (text browser state, HTML snippet, geolocation view, or screenshot), chooses action $\mathcal{A}_t \sim \pi_\theta(\cdot \mid \mathcal{O}_{1:t}, \mathcal{A}_{1:t-1})$, and the environment transitions. The task succeeds if a terminal condition is met within horizon $T$. The objective is to maximise the expected task success rate

$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \mathbb{1}\{\text{task complete}\} \right]$

across a held-out task distribution.

Explicit assumption list. Per-paper assumptions differ; the shared assumptions across the lineage:

1. Sufficient observability. The chosen observation modality (text, HTML, screenshot) carries enough information to solve the task. WebGPT assumes text browser state suffices; Computer Use and CUA assume a screenshot suffices.
2. Action vocabulary completeness. The defined action set can express any task-relevant operation. WebGPT’s six commands suffice for question-answering; Computer Use’s cursor + keyboard suffices for any GUI.
3. Reasonable horizon. Tasks terminate within a budget (WebGPT: 100 browser actions; WebAgent: ~20 steps on real-estate tasks ³ ; CUA: undisclosed but bounded).
4. Reward / verifier availability. WebGPT trains a reward model from human preferences. WebAgent uses environmental feedback (execution errors, retriever failures) to filter self-generated trajectories. CUA’s RL post-training uses an undisclosed reward signal — [Reviewer Perspective] a notable transparency gap.

Why the problem is hard. Long-horizon agency compounds errors. If a single step has 95% reliability, a 20-step task succeeds with probability $0.95^{20} \approx 0.358$. The real-estate domain in WebAgent runs ~20 steps and scored 65% completion — substantially better than the compound-error baseline, which implies the agent recovers from some step-level errors ³ . The four-year arc has not solved compound-error fragility in general; CUA’s 38.1% OSWorld figure means more than half of multi-step computer-use tasks still fail ⁹ .

LLM-based positioning. All five papers use a large language model as the policy. WebGPT finetunes GPT-3 (760M / 13B / 175B). WebAgent uses Flan-U-PaLM (540B) for code, HTML-T5 (3B) for planning. V-IRL evaluates off-the-shelf VLMs (GPT-4V, LLaVA-1.5, CogAgent, CLIP variants). Computer Use uses Claude 3.5 Sonnet. CUA uses a GPT-4o-derived model with RL post-training.

4. Motivation and gap

Real-world problem. Knowledge workers spend large fractions of their day on multi-step computer workflows that are individually trivial but collectively expensive: data lookups across SaaS systems, ticket triage, expense filing, scheduling, content moderation, customer-facing form-filling. An agent that closes these loops without human-in-the-loop intervention would be transformative. The four-year arc this review traces is the progressive narrowing of the gap between this aspiration and observable agent performance.

Existing approaches and failure modes (per paper).

• WebGPT (2021). Prior open-domain QA systems relied on retrieval-then-read pipelines that produced short answers without traceable provenance ¹ . The failure mode was unverifiable factuality. WebGPT addressed this by forcing the model to cite browsed references.
• WebAgent (2023). Prior web automation work was largely restricted to MiniWoB++ — synthetic single-page tasks with ~500-token HTML, far from real-world complexity ³ . Real websites run 7K–14K tokens of HTML per page, exceeding standard LLM context windows ³ . Single-LLM approaches achieved only 10–30% success on real-world websites ³ .
• V-IRL (2024). Prior multimodal agents were evaluated in synthetic environments (AI2-THOR, Habitat) whose visual distribution does not reflect global cultural and linguistic diversity ⁵ . V-IRL surfaced that vision models showed “subpar performance” in Lagos, Tokyo, Hong Kong, and Buenos Aires, particularly in non-English-dominant regions ⁵ .
• Computer Use (2024). Prior browser agents (including WebAgent) relied on parsing HTML, which means the agent fails when the underlying surface is not HTML (native macOS apps, browser canvases, drawing tools). The screen as universal interface side-steps this constraint ⁷ .
• CUA / Operator (2025). Prior screen-pixel agents (Anthropic’s October 2024 Computer Use) reached only 22.0% on OSWorld, far below the productivity threshold for hands-off automation ⁷ . CUA closed much of the gap on web-only tasks (87% WebVoyager) while still leaving substantial OSWorld headroom ⁹ .

Practical stakes. [External comparison] The economic value at stake is large: knowledge work automation is one of the most active enterprise-AI investment areas in 2025–2026. Pricing pressure on agent products is anchored to per-task success rates; an 87% WebVoyager number is materially different from a 22% OSWorld number in deployment-readiness terms.

Position in broader research landscape. [External comparison] The lineage sits at the intersection of three research traditions: tool-use LLMs (Toolformer, ReAct, language-model-as-tool-user broadly), reinforcement learning from human feedback (RLHF — WebGPT was an early instance), and embodied AI (V-IRL inherits this thread). Contemporary 2024–2026 work that extends the lineage: WebVoyager (live-web agents) ¹⁴ , OSWorld (full-computer benchmark) ¹³ , Mind2Web (action-prediction benchmark) ¹¹ , AgentBench (general agent evaluation), and AutoGen / smolagents / CrewAI (multi-agent frameworks).

5. Method overview

5.1 WebGPT — text browser + BC + reward model + rejection sampling

Architecture. Fine-tuned GPT-3 at three scales: 760M, 13B, 175B ² . The model is given a text-formatted browser observation containing the current question, the text of the current page at the cursor location, and a list of available actions.

Action space. Six command types ² :

1. Search queries via Microsoft Bing Web Search API.
2. Click a link.
3. Scroll (up / down).
4. Find in page.
5. Quote (extract a snippet as a reference).
6. End browse phase and generate the final answer.

The agent is capped at 100 browser actions per episode ² .

Training procedure. Four-stage ² : behaviour cloning on ~6,000 human demonstrations; reward modelling on ~21,500 pairwise comparisons; PPO against the reward model with a KL penalty; rejection sampling at inference (best-of-N against the reward model).

Design rationale. [From the paper:] The paper reports that the best variant uses BC + rejection sampling and that rejection sampling substantially outperformed PPO alone, with the 175B best-of-64 preferred 68% of the time over the BC baseline ² . Classification: [Adopted] — behaviour cloning, PPO, and reward modelling were established by 2021 [External comparison]; the [New] contribution is the text-browser action interface and the question-answering domain framing.

5.2 WebAgent — three-module decomposition with HTML-T5 + Flan-U-PaLM

Architecture (Figure 3 of paper). Three modules ⁴ :

1. Planner (HTML-T5). Reads the user instruction + sub-instruction history + raw HTML and predicts the next sub-instruction.
2. Summariser (HTML-T5). Reads the raw HTML and the predicted sub-instruction and emits a data-ref snippet pointing at the relevant DOM subtree, extracted via XPath.
3. Program synthesiser (Flan-U-PaLM, 540B). Reads the sub-instruction (treated as a code comment) plus the extracted HTML snippet plus few-shot examples and emits an executable Python Selenium script.

HTML-T5 architecture (Figure 4 of paper). Encoder-decoder transformer with local + transient-global attention in the encoder ⁴ :

• Local attention: each token attends only to a window of nearby neighbours (the HTML leaf-element pattern).
• Transient global attention: a small set of summary tokens computed by summation / normalisation across blocks, giving every token a global view at $O(n)$ cost.
• Decoder: dense T5-style attention.
• Sequence lengths: 4,096 input / 910 output during pretraining; 16K context window downstream ⁴ .

Pretraining objective. Mixture-of-long-span denoising. Standard T5 uses mean span length $\mu = 3$. HTML-T5 uses $\mu \in \{8, 64\}$ ⁴ .

Pretraining corpus. 100 WARC files from CommonCrawl (April 2019), preprocessed to 3.41M examples; extraction focused on subtrees around <label> elements with for attributes to reduce noise ⁴ .

Self-experience supervision. Scripted agents procedurally parse instructions into sub-instructions; regex-based retrieval identifies elements; Flan-U-PaLM executes web actions; environmental feedback (execution errors, retriever failures, malformed URLs) filters demonstrations; HTML-T5 finetunes on successful episodes ⁴ .

Classification. [New]: HTML-T5 architecture, mixture-of-long-span denoising objective for HTML, three-module decomposition. [Adopted]: T5 encoder-decoder skeleton, LongT5’s local + transient-global attention, Flan-U-PaLM as program synthesiser. [Adapted]: Self-experience supervision is adapted from RL-from-self-play traditions to a filtered demonstration setting.

5.3 V-IRL — geolocation-grounded multi-agent platform

Platform architecture. Hierarchical four-layer design ⁶ :

1. Environment (action). Geolocation-based navigation through Google Maps Platform; street view imagery with adjustable heading, pitch, FOV; routing and place-type metadata.
2. Vision (perception). Off-the-shelf vision models for object detection, localisation, and VQA (CLIP variants, GroundingDINO, GLIP, Owl-ViT, LLaVA, BLIP-2, CogAgent, GPT-4V).
3. Language (reasoning). LLM-based decision-making and planning.
4. Collaboration. Agent-to-agent and human-agent coordination scaffolding.

Nine exemplar agents. Each agent demonstrates a capability slice ⁶ : Peng (route optimisation), Aria (recommendation), RX-399 (object detection / counting), Hiro (iterative exploration), Ling (VLN), Diego (itinerary planning), plus three others.

Three benchmarks introduced. [From the paper:] ⁶

• V-IRL Place Localisation: 20 place categories from street view; best is CLIP with GLIP proposal at 24.6% average recall.
• V-IRL Place Recognition + VQA: 96 place types; BLIP-2 leads VQA at 69.6%, CLIP (L/14@336px) leads recognition at 41.3%.
• V-IRL Vision-Language Navigation: end-to-end navigation following natural-language directions. CLIP (L/14@336px) achieves 44% success vs. 100% oracle.

Classification. [New]: the V-IRL platform itself, the three geolocation-grounded benchmarks, the cultural / linguistic bias finding. [Adopted]: the off-the-shelf vision and language models used; Google Maps Platform APIs.

5.4 Anthropic Computer Use — screen + cursor + keyboard

Architecture. Claude 3.5 Sonnet (October 2024 upgrade) with three tool primitives ⁷ :

1. Screenshot. The model receives a rendered image of the current screen.
2. Mouse. The model emits a coordinate to click, double-click, or right-click.
3. Keyboard. The model emits a key sequence to type.

Inference loop. Observe screenshot, reason, emit a tool call (mouse / keyboard / take-screenshot), repeat. The model decides when the task is complete.

Training approach. Not publicly disclosed. The Model Card Addendum ⁸ documents the evaluation results but does not specify the training recipe. [Reviewer Perspective] This is a transparency gap relative to the academic papers in the lineage; readers cannot reproduce or extend the training pipeline.

Safety mitigations. New classifiers identify when computer use is being deployed and whether harm is occurring; specific threats identified include spam, misinformation, and fraud ⁷ .

Classification. [New]: the screenshot-plus-coordinate action surface, the public-API release of computer-use as a developer primitive. [Adopted]: the underlying Claude 3.5 Sonnet base model.

5.5 OpenAI Operator / CUA — screenshot + GPT-4o + RL reasoning

Architecture. GPT-4o vision capabilities plus reasoning trained with reinforcement learning ⁹ . Single general action space: cursor click, scroll, type, screenshot.

Inference loop. Same screenshot-plus-action pattern as Computer Use; the differentiator is the underlying model strength and the RL post-training.

Training approach. [From the paper:] “combines GPT-4o’s vision capabilities with advanced reasoning through reinforcement learning” ⁹ . The specific reward signal and training data are not publicly disclosed in the launch communication or system card. [Reviewer Perspective] Same transparency gap as Computer Use.

Safety mitigations. Multi-layered defences against prompt injection including a separate classifier that monitors screenshots for adversarial content ¹⁰ . Risk identification process evaluated both user goals (“tasks”) and the steps the model takes (“actions”).

Classification. [New]: the specific RL-on-reasoning post-training recipe (not publicly described), the OSWorld 38.1% / WebVoyager 87% benchmark headlines. [Adopted]: the screenshot-and-cursor interface from Computer Use; the GPT-4o base.

6. Mathematical contributions

The five papers are not heavily mathematical relative to, say, optimisation or theoretical-ML papers. The math that does exist concentrates in WebGPT’s training objective and WebAgent’s pretraining objective and attention pattern. This section reproduces the load-bearing formulas with worked examples.

MATH ENTRY 1: WebGPT BC + RM + PPO objective

• Source: WebGPT Section 3 (Training) ² .
• What it is: [Reconstructed] The composite training pipeline. WebGPT trains in four stages; the reconstructed formal objectives below follow standard BC + RM + PPO conventions because the paper describes the pipeline procedurally rather than with explicit equations.
• Formal definition (BC stage).

$\mathcal{L}_{\text{BC}}(\theta) = -\mathbb{E}_{(s, a) \sim \mathcal{D}_{\text{demo}}} \left[ \log \pi_\theta(a \mid s) \right]$

• Formal definition (reward-model stage).

$\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(s, a_w, a_l) \sim \mathcal{D}_{\text{pref}}} \left[ \log \sigma\!\left( r_\phi(s, a_w) - r_\phi(s, a_l) \right) \right]$

• Formal definition (PPO stage).

$J_{\text{PPO}}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ r_\phi(s, a) - \beta \, \mathrm{KL}\!\left( \pi_\theta \,\|\, \pi_{\text{ref}} \right) \right]$

• Each term explained and dimensional analysis.

  • $s$: the text-formatted browser observation at one step (a string of up to a few thousand tokens).
  • $a$: an action token sequence (a browser command like search[query]).
  • $\pi_\theta(a \mid s)$: probability that the policy emits action $a$ given state $s$ (a scalar in $[0, 1]$).
  • $r_\phi(s, a)$: the reward model’s scalar evaluation of state-action pair (a single real number).
  • $\sigma(\cdot)$: sigmoid function $\sigma(x) = 1 / (1 + e^{-x})$, mapping a real to $[0, 1]$.
  • $a_w, a_l$: winning and losing action in a human preference pair.
  • $\beta$: KL-penalty coefficient (a positive scalar, hyperparameter).
  • $\pi_{\text{ref}}$: reference policy (typically the BC checkpoint), used to prevent drift.

• Worked numerical example (BC stage). Suppose a single training example: state $s$ contains the question “What causes thunderstorms?” plus the current browser page; demonstrated action is search[atmospheric instability causes thunderstorms]. Tokenise the action to $K = 6$ tokens. The BC loss for this example is $\ell = -\sum_{k=1}^{6} \log \pi_\theta(a_k \mid s, a_{<k}) = -(\log 0.4 + \log 0.6 + \log 0.3 + \log 0.5 + \log 0.7 + \log 0.4) \approx 5.18$ if the per-token probabilities are 0.4, 0.6, 0.3, 0.5, 0.7, 0.4 respectively. The optimiser steps $\theta$ to reduce this loss.

• Worked numerical example (RM stage). Suppose two candidate answers $a_w$ (preferred) and $a_l$ (not preferred). Suppose $r_\phi(s, a_w) = 2.1$ and $r_\phi(s, a_l) = 0.4$. Then $\sigma(2.1 - 0.4) = \sigma(1.7) = \frac{1}{1 + e^{-1.7}} \approx 0.846$ and the RM loss for this pair is $-\log 0.846 \approx 0.167$. Loss decreases as the reward gap grows.

• Worked numerical example (rejection sampling at inference). With the 175B model and best-of-64, generate 64 candidate answers $\{a^{(1)}, \ldots, a^{(64)}\}$, score each by $r_\phi(s, a^{(i)})$, return the argmax. Suppose the scores are uniformly distributed in $[-1, 3]$; the expected top-of-64 score under a uniform model is approximately $3 - 4/(64+1) \approx 2.94$, while the mean is $1.0$. Best-of-64 yields a substantial quality lift over a single sample ² .

• Role. Trains the policy to maximise expected reward while staying close to the reference policy. Rejection sampling at inference further amplifies quality without additional RL.

• Edge cases. When the reward model is mis-specified, PPO reward-hacks (policy drifts to high-reward but low-quality outputs); the KL penalty mitigates but does not eliminate this.

• Novelty. [Adopted]: BC, PPO, reward modelling were established 2017–2020 [External comparison]. [New]: the text-browser instantiation and the ELI5 domain.

• Transferability. [Analysis] The four-stage pipeline transfers to any agent-RL setting with human-preference labels; it is the canonical RLHF recipe and ships unchanged in InstructGPT, Sparrow, and modern post-training stacks.

• Why it matters. This is the training recipe that established RLHF as the default agent-alignment approach.

MATH ENTRY 2: HTML-T5 mixture-of-long-span denoising

• Source: WebAgent Section 3.2 (HTML-T5 pretraining) ⁴ .
• What it is: A pretraining objective that masks contiguous chunks of HTML text and asks the model to predict them, with mean span length drawn from a longer distribution than standard T5.
• Formal definition. For an HTML document tokenised to $\mathbf{x} = (x_1, \ldots, x_n)$, sample a span length $L \sim \mathrm{Geom}(1/\mu)$ for each masked region, mask 15% of input tokens, and train

$\mathcal{L}_{\text{denoise}}(\theta) = -\mathbb{E}_{\mathbf{x}, \mathcal{M}} \left[ \log p_\theta(\mathbf{x}_{\mathcal{M}} \mid \mathbf{x}_{\setminus \mathcal{M}}) \right]$

where $\mathcal{M}$ is the set of masked positions sampled with mean span length $\mu \in \{8, 64\}$ (mixture) ⁴ .

• Each term explained and dimensional analysis.

  • $\mathbf{x}$: an HTML token sequence of length $n$ (up to 4,096 during pretraining).
  • $\mathcal{M} \subseteq \{1, \ldots, n\}$: a subset of positions to mask, $\mid \mathcal{M}\mid \approx 0.15 n$.
  • $\mathbf{x}_\mathcal{M}$: the masked tokens (target).
  • $\mathbf{x}_{\setminus \mathcal{M}}$: the unmasked tokens (encoder input).
  • $\mu$: mean span length parameter; the distribution mixes $\mu = 8$ and $\mu = 64$.
  • $\mathrm{Geom}(1/\mu)$: geometric distribution over span lengths with mean $\mu$.

• Worked numerical example (small scale). Take an HTML fragment <form class="login" action="/api/submit" method="post"><input id=. Tokenise to 16 tokens (assume one token per word / punctuation for simplicity). Mask 15% gives ~2 tokens. With $\mu = 3$ (standard T5), expect spans of length ~3, but for a 2-token mask budget the result is one masked span of length 2 or 3 covering syntactic tokens like class=" or id=. With $\mu = 8$, sample $L = 7$ from $\mathrm{Geom}(1/8)$; the masked span is more likely to be a semantically meaningful chunk like <form class="login" action= — the model now has to recover content, not just syntax.

• Worked numerical example (mixture). The mixture $\mu \in \{8, 64\}$ samples either an 8-token or a 64-token mean span per masking event. For a 4,096-token HTML input, a 64-token span covers a whole tag block (e.g., a complete <div>...</div> subtree). The mixture forces the encoder to be useful at multiple granularities — local tag-level and global subtree-level.

• Role. Replaces the standard T5 pretraining objective for HTML domains. Produces an encoder that captures HTML semantics rather than just HTML syntax.

• Edge cases. If $\mu$ is too large relative to $n$, masking can swallow whole documents and training collapses. The paper’s $\mu \in \{8, 64\}$ is empirically tuned for $n = 4{,}096$.

• Novelty. [Adapted]: span denoising is the original T5 recipe [External comparison]; the [New] contribution is the long-mixture distribution tuned for HTML.

• Transferability. [Analysis] The recipe transfers to any structured-text domain where syntax tokens shouldn’t dominate the masking signal — XML, source code, configuration files, log streams. Worth borrowing for any vertical-LLM pretraining where the corpus has heavy syntactic regularity.

• Why it matters. [From the paper:] Ablation shows HTML-denoising with $\mu \in \{8, 64\}$ gives 57.0% MiniWoB++ vs. 53.8% without HTML-denoising and 54.1–61.1% with instruction-finetuning alone ⁴ .

MATH ENTRY 3: Local + transient-global attention

• Source: WebAgent Section 3.2 (HTML-T5 architecture), Figure 4 ⁴ .
• What it is: An attention pattern that combines local (windowed) attention with a small set of summary tokens, reducing per-layer cost from $O(n^2)$ to roughly $O(n \cdot w + n \cdot g)$ where $w$ is the local window and $g$ is the number of global tokens.
• Formal definition. Given queries $\mathbf{Q} \in \mathbb{R}^{n \times d}$, keys $\mathbf{K}$, values $\mathbf{V}$, partition the sequence into blocks of size $B$. Compute global tokens $\mathbf{G} \in \mathbb{R}^{(n/B) \times d}$ by summing and normalising tokens within each block. Then for token $i$:

$\mathbf{h}_i = \mathrm{softmax}\!\left( \frac{\mathbf{Q}_i [\mathbf{K}_{\text{local}(i)}, \mathbf{K}_G]^\top}{\sqrt{d}} \right) [\mathbf{V}_{\text{local}(i)}, \mathbf{V}_G]$

where $\mathbf{K}_{\text{local}(i)}$ are the keys in the $w$-token window around $i$ and $\mathbf{K}_G$ are the global summary keys.

• Each term explained and dimensional analysis.

  • $\mathbf{Q}_i \in \mathbb{R}^d$: the query vector at position $i$.
  • $\mathbf{K}_{\text{local}(i)} \in \mathbb{R}^{w \times d}$: $w$ neighbouring keys.
  • $\mathbf{K}_G \in \mathbb{R}^{(n/B) \times d}$: global summary keys; one per block of size $B$.
  • $\mathbf{V}$ analogous to $\mathbf{K}$.
  • $\sqrt{d}$: standard attention temperature.
  • $\mathbf{h}_i \in \mathbb{R}^d$: the output representation at position $i$.

• Worked numerical example. Take $n = 4{,}096$, $d = 768$, $w = 128$ local window, $B = 64$ block size (so $n / B = 64$ global tokens). Standard dense attention per token attends to all $n = 4{,}096$ keys, cost $O(n)$ per token, $O(n^2) = 16.8$M operations total. Local + global attention per token attends to $w + n/B = 128 + 64 = 192$ keys, cost $O(192)$ per token, $O(n \cdot 192) = 786{,}432$ operations total. 21× reduction in attention cost at $n = 4{,}096$, growing as $n$ grows.

• Role. Lets HTML-T5 process 4K-token HTML documents at training time and 16K-token contexts downstream without quadratic memory blowup.

• Edge cases. Information that must travel between distant local windows passes through global tokens; if the task requires fine-grained cross-window attention, the bottleneck shows. The paper’s ablation shows local + global beats dense by 18 percentage points on MiniWoB++ at 12K training examples ⁴ , suggesting the bottleneck is not severe for HTML.

• Novelty. [Adopted]: the local + transient-global pattern is from LongT5 [External comparison]. The [New] application is to HTML.

• Transferability. [Analysis] Any long-context structured-text encoder can adopt the pattern; modern long-context models (Mamba, Hyena, sliding-window Transformers) explore similar locality biases.

• Why it matters. The 18 percentage-point lift over dense attention on the same training budget is the evidence that long-context architecture matters for HTML-aware agents.

MATH ENTRY 4: Compound-error model for long-horizon agents

• Source: [Analysis] — not explicit in any single paper but implied across all five.
• What it is: A simple model of why long-horizon agents fail. If each step succeeds independently with probability $p$, a $T$-step task succeeds with probability $p^T$.
• Formal definition.

$P(\text{task success}) = p^T$

assuming step-level independence and no recovery from errors.

• Worked numerical example. Take WebAgent’s real-estate domain at $T = 20$ steps ⁴ . Observed success rate is 65%. Solving $p^{20} = 0.65$ gives $p = 0.65^{1/20} \approx 0.979$. Step-level reliability ~97.9% is required to hit 65% over 20 steps under the independence assumption. If steps are correlated (later steps depend on early ones being right), required step-level reliability is higher.
• Worked numerical example (CUA OSWorld). At 38.1% success rate on OSWorld ⁹ , and assuming an average $T = 15$ steps, the implied step-level reliability is $p = 0.381^{1/15} \approx 0.938$. CUA’s underlying step-level decisions are roughly 94% reliable on average; the residual 6% per-step error compounds over 15 steps into a 62% task failure rate.
• Role. [Analysis] Explains why benchmark numbers in the 30–60% range are consistent with very high (>90%) step-level reliability; the multiplicative compounding is the dominant force.
• Edge cases. Real agents recover from some errors (CUA self-corrects per OpenAI’s description ⁹ ), so the bare $p^T$ underestimates actual reliability — but it gives the right intuition for why long horizons are hard.
• Why it matters. [Analysis] Practitioners reading benchmark numbers in the 30–60% range often dismiss them as low; the compound-error lens shows these numbers are consistent with high underlying step reliability, and that incremental step-reliability gains translate to large task-success gains.

7. Algorithmic contributions

The two papers in the lineage with explicit algorithmic pseudocode are WebGPT (training loop) and WebAgent (inference loop). V-IRL, Computer Use, and CUA describe their pipelines procedurally without published pseudocode.

ALGORITHM ENTRY 1: WebGPT four-stage training loop

• Source: WebGPT Section 3 ² .
• Purpose: Train a browser-using question-answering policy.
• Inputs: Pretrained GPT-3 (760M / 13B / 175B); 6K human demonstrations $\mathcal{D}_{\text{demo}}$; 21.5K human comparisons $\mathcal{D}_{\text{pref}}$; browser environment $\mathcal{E}$ with six commands.
• Outputs: Trained policy $\pi_\theta$.
• Pseudocode (reconstructed).

def webgpt_train(theta_0, D_demo, D_pref, env, N_samples):
    # Stage 1: Behaviour cloning
    theta = theta_0
    for (s, a) in D_demo:
        theta = sgd_step(theta, loss=-log pi_theta(a | s))

    # Stage 2: Reward modelling
    phi = init_reward_model()
    for (s, a_w, a_l) in D_pref:
        phi = sgd_step(phi, loss=-log sigmoid(r_phi(s, a_w) - r_phi(s, a_l)))

    # Stage 3: PPO against r_phi
    for episode in range(N_episodes):
        tau = rollout(pi_theta, env)
        r = r_phi(tau)
        theta = ppo_step(theta, tau, r, kl_penalty=beta, ref=theta_bc)

    # Stage 4: Inference-time rejection sampling
    def inference(s):
        candidates = [sample(pi_theta, s) for _ in range(N_samples)]
        return argmax(candidates, key=lambda a: r_phi(s, a))

    return pi_theta, r_phi, inference

• Hand-traced example. Take the question “What causes thunderstorms?”. Stage 1 BC gives policy $\pi_\theta$ that emits search[thunderstorm causes] with probability ~0.4 after demonstration finetuning. Stage 2 RM scores two candidate final answers, one with citations and one without; the cited one scores higher. Stage 3 PPO updates $\theta$ to favour rollouts the RM scores highly — the policy now emits cite-rich answers. Stage 4 inference: generate 64 answer candidates, pick the one with highest $r_\phi$ score.
• Complexity. Time: dominated by PPO (linear in number of episodes × rollout length × forward-backward cost). Space: held in $O(\text{model parameters})$. Bottleneck step: PPO rollouts at 175B scale.
• Hyperparameters. BC: standard supervised loss, batch size and LR per paper Appendix. RM: same. PPO: $\beta$ (KL coefficient), learning rate, episode budget. Rejection sampling: $N = 4 / 16 / 64$ for 760M / 13B / 175B ² .
• Failure modes. PPO can reward-hack if the reward model is mis-specified; the paper notes this and uses rejection sampling as a more reliable inference-time alternative.
• Novelty. [Adopted]: the four-stage pipeline is the canonical RLHF recipe established 2019–2021 [External comparison]. [New]: text-browser environment, browser action space, ELI5 domain.
• Transferability. [Analysis] Direct lift to any tool-using RLHF setting.

ALGORITHM ENTRY 2: WebAgent inference loop

• Source: WebAgent Section 3.1, Figure 5 episode trace ⁴ .
• Purpose: Close a real-world web task end-to-end through plan-summarise-synthesise loop.
• Inputs: User instruction $u$; raw HTML of current page $h$; sub-instruction history $H$; HTML-T5 planner $\pi_{\text{plan}}$; HTML-T5 summariser $\pi_{\text{sum}}$; Flan-U-PaLM synthesiser $\pi_{\text{prog}}$; browser env (Selenium WebDriver).
• Outputs: Successful task completion or step-budget exhaustion.
• Pseudocode (reconstructed from procedural description).

def webagent_run(u, env, max_steps=20):
    H = []
    for step in range(max_steps):
        h = env.get_html()
        # Plan: predict next sub-instruction
        sub = pi_plan(u, H, h)
        if sub == "DONE":
            return success(env)
        # Summarise: extract relevant HTML snippet via XPath
        data_ref = pi_sum(sub, h)
        snippet = xpath_extract(h, data_ref)
        # Synthesise: emit Python Selenium script
        code = pi_prog(sub, snippet, few_shot_examples)
        try:
            exec(code, env=env)
        except (ExecutionError, RetrieverError, MalformedURL):
            return failure("execution error")
        H.append(sub)
    return failure("step budget exhausted")

• Hand-traced example. Task: “Find a 3-bedroom apartment in Brooklyn under $4K/month”. Step 1: $\pi_{\text{plan}}$ reads $u$ + empty history + Zillow HTML, emits sub-instruction enter "Brooklyn" in location search. $\pi_{\text{sum}}$ reads sub + HTML, emits data-ref="search-location-input". XPath extracts the snippet <input id="search-location-input" type="text">. $\pi_{\text{prog}}$ reads sub + snippet, emits driver.find_element(By.ID, "search-location-input").send_keys("Brooklyn"). Execute. Step 2: sub = apply filter: bedrooms = 3. And so on, for ~20 steps until done. If any execution errors, the agent fails the task ⁴ .
• Complexity. Time: dominated by Flan-U-PaLM (540B) calls per step. Space: HTML can run 7K–14K tokens ⁴ , but XPath extraction reduces the snippet handed to the synthesiser. Bottleneck: program synthesis call at 540B scale.
• Hyperparameters. Max steps: ~20 for real-estate domain. Few-shot example count: not specified; paper Appendix.
• Failure modes. Error breakdown on real websites ⁴ : real-estate 20% programming, 33% planning, 25% summarisation; social-media 70% programming, 50% planning, 50% summarisation; map 10% programming, 17% planning, 25% summarisation. Planning failures (sub-instruction conflicts with user goal) are the fundamental bottleneck for long horizons.
• Novelty. [New]: the three-module decomposition. [Adopted]: Selenium WebDriver, XPath.
• Transferability. [Analysis] The plan-summarise-synthesise pattern transfers to any structured-environment agent task; the specific modules and their training data are domain-specific.

ALGORITHM ENTRY 3: Computer Use / CUA inference loop

• Source: [Reconstructed] from Anthropic Computer Use announcement ⁷ and OpenAI CUA description ⁹ .
• Purpose: Close a computer-use task by observing screenshots and emitting mouse/keyboard actions.
• Inputs: User task description $u$; OS / browser environment $\mathcal{E}$; model $\pi$ (Claude 3.5 Sonnet or CUA).
• Outputs: Task completion or step-budget exhaustion.
• Pseudocode (reconstructed from procedural description).

def computer_use_run(u, env, max_steps):
    history = []
    while not done(history) and len(history) < max_steps:
        img = env.screenshot()
        action = pi(u, history, img)
        # action is one of:
        #   click(x, y) | double_click(x, y) | right_click(x, y)
        #   type(text) | key(key_code)
        #   scroll(direction) | screenshot()
        #   done()
        if action == done():
            return success(env)
        env.execute(action)
        history.append((img, action))
    return failure("step budget exhausted")

• Hand-traced example. Task: “Book a flight from NYC to SFO for next Tuesday on United”. Step 1: screenshot of empty desktop. Model emits click(450, 800) to click the Chrome icon. Step 2: screenshot of Chrome opening. Model emits click(720, 100) on the URL bar, then type("united.com"), then key(Return). Step 3: screenshot of united.com homepage. Model emits clicks on departure city, destination city, date picker. And so on for many steps. Each step is a screenshot-plus-action; the agent self-corrects when it sees the wrong outcome on the next screenshot.
• Complexity. Time: dominated by per-step model inference (vision-language forward pass on a screenshot plus history). Space: history can grow large; modern context windows (200K+ tokens) accommodate dozens of screenshots.
• Hyperparameters. Max steps: not publicly disclosed for either Computer Use or CUA. Action vocabulary: cursor primitives, keyboard, screenshot, done.
• Failure modes. OSWorld 14.9% / 22.0% (Computer Use) ⁸ and 38.1% (CUA) ⁹ imply 60–85% task failure on full computer-use tasks. Common failure modes [Analysis]: misclicks on small UI elements, OCR failures on complex strings (DNA sequences, API keys, crypto wallet addresses) ⁹ , getting stuck in dialog loops, prompt injection from page content.
• Novelty. [New]: the unified screenshot-plus-action interface as a developer primitive; the OSWorld / WebArena / WebVoyager benchmark dominance.
• Transferability. [Analysis] The interface is the durable contribution; any future computer-use model can adopt it. The model weights are proprietary.

8. Specialised design contributions

Subsection 8A — LLM / prompt design

WebGPT, WebAgent, Computer Use, and CUA all involve carefully designed prompts wrapping the model’s observation. The prompts are not published verbatim for any of the four; the structural patterns can be reconstructed.

PROMPT ENTRY 1: WebGPT browser observation.

• Source: WebGPT Section 2 ^{1 2} .
• Role: The text-formatted browser state handed to GPT-3 at each step.
• Prompt type: Few-shot context-conditioned generation.
• Components in order (reconstructed): (1) the current question; (2) the text of the current page at cursor location; (3) recent action history; (4) available action commands and their syntax; (5) a request to emit the next action.
• Input schema: plain-text observation. Output schema: a single browser command in the defined six-command vocabulary.
• Novelty. [New]: the text-browser observation format. [Adopted]: few-shot prompting context.

PROMPT ENTRY 2: WebAgent program-synthesis prompt.

• Source: WebAgent Section 3.1 ⁴ .
• Role: Hands the predicted sub-instruction and extracted HTML snippet to Flan-U-PaLM, which emits a Python Selenium script.
• Components in order (reconstructed): (1) few-shot examples of (sub-instruction, snippet, Python code) triples; (2) the sub-instruction (treated as a code comment in the prompt); (3) the extracted HTML snippet; (4) a prompt suffix asking for the Python action.
• Output schema: a Python expression using Selenium’s driver.find_element(...) API.
• Novelty. [Adapted]: program synthesis from natural-language plus snippet is a Codex-era pattern [External comparison].

Subsection 8B — Architecture-specific details

Covered in Section 5 per paper. The HTML-T5 local + global attention (Section 5.2, MATH ENTRY 3) is the most notable architecture-specific contribution in the lineage.

Subsection 8C — Training specifics

WebGPT: Hardware not disclosed in detail. Batch size, LR, optimiser per Appendix; the paper notes shared hyperparameters across model sizes with Adam step size adjustments ² .

WebAgent / HTML-T5: Pretrained on 100 WARC files from CommonCrawl April 2019 (3.41M examples); finetuned on 347K demonstrations for MiniWoB++ and ~900 episodes across 3 real-world websites for the production WebAgent ⁴ . Initialised from PEGASUS (the paper notes this is critical for downstream performance ⁴ ).

V-IRL: No model training; evaluates off-the-shelf models ⁶ .

Computer Use, CUA: Training specifics not publicly disclosed ^{7 9} .

Subsection 8D — Inference / deployment specifics

WebGPT: Rejection sampling at inference (best-of-4 / -16 / -64 for the three model sizes) ² .

WebAgent: Step budget ~20 for real-estate; XPath extraction reduces synthesiser input from 7K–14K tokens of HTML to a focused snippet ⁴ .

V-IRL: Off-the-shelf model inference through Google Maps Platform APIs ⁶ .

Computer Use, CUA: Screenshot-plus-action loop; step budget not disclosed. CUA uses a separate prompt-injection classifier monitoring screenshots ¹⁰ .

9. Experiments and results

9.1 Datasets and benchmarks

WebGPT: ELI5 (Reddit-sourced long-form QA) and TruthfulQA ^{1 2} .

WebAgent: MiniWoB++ (56 simulated tasks), Mind2Web (137 real websites across 31 domains), WebSRC (web structural reading comprehension), three production real-world websites (real-estate, social-media, map) ^{3 4} .

V-IRL: Three internally-introduced benchmarks (V-IRL Place Localisation, V-IRL Place Recognition + VQA, V-IRL Vision-Language Navigation) spanning 14 districts across 12 cities ^{5 6} .

Computer Use: OSWorld ^{7 8} .

CUA / Operator: OSWorld, WebArena, WebVoyager ^{9 10} .

9.2 Baselines and metrics

[External comparison] The lineage’s evaluation registers cluster across the four canonical benchmark families:

Benchmark	Scoring	Coverage	Cited in
MiniWoB++	Task success rate (56 simulated tasks)	Single-page synthetic	WebAgent 3
Mind2Web	Step success / element accuracy / operation F1	Offline action prediction, 137 real sites	WebAgent 11
WebArena	Task success (self-hosted realistic sites)	E-commerce, CMS, forums	CUA 12
WebVoyager	Task success (live sites: Amazon, GitHub, Google Maps)	Real live web	CUA 14
OSWorld	Task success (full computer use)	Ubuntu, Windows, macOS	Computer Use, CUA 13

9.3 Key quantitative results

WebGPT (ELI5) ^{1 2} . Best 175B model preferred 56% over human demonstrators; preferred 69% over top Reddit responses; 54% on TruthfulQA (truthful and informative). 175B best-of-64 preferred 68% over BC baseline.

WebAgent (real-world websites) ^{3 4} . Real-estate 65% success / 87.6% score; social-media 70% success / 85.8% score; map 80% success / 93.8% score. Single Flan-U-PaLM without modular decomposition: 10–30% success.

WebAgent (Mind2Web) ⁴ . HTML-T5-XL: element accuracy 76.4% (vs. MindAct 55.1%); operation F1 78.8% (vs. 75.7%); step success rate 71.5% (vs. 52.0%).

WebAgent (MiniWoB++) ⁴ . HTML-T5-XL 67.1% (18.7-point improvement over prior best WebN-T5 at 48.4%); with 347K demonstrations, 85.6% — exceeding Flan-T5-XXL (11B) despite HTML-T5-XL being only 3B parameters.

V-IRL ^{5 6} . Place Localisation best 24.6% (CLIP + GLIP). Place Recognition best 41.3% (CLIP L/14@336px). VQA best 69.6% (BLIP-2). VLN best 44% (CLIP L/14@336px) vs. 100% oracle.

Computer Use ⁷ . OSWorld 14.9% (single screenshot), 22.0% (with additional steps).

CUA / Operator ⁹ . OSWorld 38.1%, WebArena 58.1%, WebVoyager 87%.

9.4 Ablations (WebAgent — most thorough)

Architecture comparison on MiniWoB++ with 12K examples ⁴ : Flan-T5-Base dense 35.3%; LongT5-Base local 44.0%; LongT5-Base local + global 53.6% (+18 points over dense).

Pretraining-objective ablation ⁴ : HTML-denoising $\mu \in \{8, 64\}$ 82.46 real-estate accuracy, 57.0% MiniWoB++; instruction-finetuning alone (Flan-LongT5) 54.1–61.1% MiniWoB++; no HTML-denoising 53.8% MiniWoB++.

Dataset preprocessing ⁴ : extracted subtrees 82.46 accuracy; raw HTML 80.56 accuracy. PEGASUS initialisation flagged as critical.

9.5 Independent benchmark cross-checks for SOTA claims

[Analysis] The SOTA claims in the lineage face partial reproducibility coverage:

• WebGPT. Behaviour-cloning + reward-modelling + PPO + rejection sampling has been independently replicated (InstructGPT, Sparrow). The ELI5 human-preference numbers are the original paper’s framing on the chosen task; independent reproduction on the same task set has not been widely published.
• WebAgent. Mind2Web numbers are part of the published Mind2Web benchmark ¹¹ ; the 71.5% step success rate is reproducible by anyone running HTML-T5-XL against the Mind2Web test set. The real-world website numbers (65–80% completion) are on the paper’s three internal websites and have not been independently reproduced.
• V-IRL. All three benchmarks are introduced in the paper itself; cross-paper independent reproduction does not yet exist.
• Computer Use and CUA. OSWorld is an independent benchmark ¹³ ; the 38.1% (CUA) and 22.0% (Computer Use) numbers can in principle be cross-checked by independent groups, though access to the vendor models is gated. WebArena and WebVoyager are similarly independent benchmarks ^{12 14} .

[Reviewer Perspective] The CUA WebVoyager 87% number specifically should be read against the WebVoyager paper’s own framing of what counts as success and what the human baseline is; the 100% oracle assumption in WebVoyager is itself a benchmark-design choice that affects how to read “matched human performance” ¹⁴ .

9.6 Evidence audit

Strongly supported claims [Analysis]: WebAgent’s modular-vs-monolithic gain (50+ percentage points on real websites) is supported by the ablation in Table 1 ⁴ . CUA’s OSWorld / WebArena / WebVoyager numbers are supported by the cited launch communication ⁹ and reproducible in principle.

Partially supported claims: Computer Use’s specific training recipe is not disclosed; the OSWorld numbers are reported but the methodology behind them (which model variant, which inference scaffold) is not fully described in the launch announcement ⁷ .

Claims relying on narrow evidence: V-IRL’s geographic-bias finding rests on the paper’s 14-region sample; the directional finding (vision models perform worse in non-English-dominant regions) is likely directionally correct but the magnitude could change with broader sampling ⁶ .

10. Technical novelty summary

Component	Type	Novelty level	Justification	Paper
Text-browser action interface	Architecture	Incrementally novel	First instance of a general text-formatted browser for LLM-as-agent	WebGPT
BC + RM + PPO + rejection sampling	Training	Combination novel	Components individually adopted; the combination established RLHF for agents	WebGPT
Three-module decomposition (plan/summarise/synthesise)	Architecture	Fully novel	The specific WebAgent split	WebAgent
HTML-T5 architecture	Architecture	Combination novel	LongT5 attention adapted for HTML	WebAgent
Mixture-of-long-span denoising for HTML	Training	Incrementally novel	T5 span denoising tuned for HTML	WebAgent
Self-experience supervision	Training	Adapted	Inspired by self-play RL traditions	WebAgent
Geolocation-grounded agent platform	System	Fully novel	The V-IRL platform itself	V-IRL
Cultural / linguistic vision-model bias finding	Empirical	Fully novel	First systematic surfacing of this failure mode in agents	V-IRL
Screenshot + cursor as universal interface	Architecture	Incrementally novel	Visible in prior research (SeeClick, CogAgent) [External comparison]; productionised by Anthropic	Computer Use
RL on reasoning for screen-pixel agents	Training	Fully novel (recipe undisclosed)	The CUA-specific training recipe	CUA
Multi-layered prompt-injection defence including screenshot classifier	Safety	Fully novel (deployment)	First production system documenting this defence	CUA

Single most novel contribution. [Analysis] WebAgent’s three-module decomposition is the most enduring technical contribution in the lineage. The plan-summarise-synthesise pattern is the architectural default for any non-trivial workflow agent in 2026, and the 50-point lift over monolithic prompting is the strongest empirical evidence in the lineage that decomposition matters more than monolithic-model scaling for long-horizon tasks.

What the papers do NOT claim to be novel. WebGPT does not claim BC, PPO, or reward modelling are novel — only their composition and the browser instantiation. WebAgent does not claim Selenium WebDriver, XPath, or T5 are novel. V-IRL does not claim any of the off-the-shelf vision models or Google Maps APIs are novel. Computer Use and CUA do not publicly claim the underlying base model architectures are novel beyond what was previously released.

11. Situating the work

What prior work did. [External comparison] Pre-WebGPT browser-assisted QA was retrieval-then-read, with no traceable citations. Pre-WebAgent web agents (WebShop, WebN-T5) scored ~48.4% on MiniWoB++ and were untested on real websites ⁴ . Pre-V-IRL multimodal agents (AI2-THOR, Habitat, ALFRED) lived in synthetic environments. Pre-Computer Use, screen-pixel agents (SeeClick, CogAgent) were research artefacts; the production API was not available.

What these papers changed conceptually. The lineage tells a coherent story: the interface narrowed (text browser → HTML-DOM → geolocation-grounded → screen pixels), the action vocabulary converged (six commands → three modules emitting Python → mouse + keyboard), and the reasoning got stronger underneath (GPT-3 → Flan-U-PaLM → GPT-4o + RL).

Contemporaneous related papers.

• Mind2Web (arXiv:2306.06070, May 2023) ¹¹ . Concurrent with WebAgent. Where WebAgent proposes a method, Mind2Web proposes the canonical benchmark for real-world action prediction. WebAgent’s strongest external validation is its 71.5% step success rate on Mind2Web ⁴ .
• WebArena (arXiv:2307.13854, July 2023) ¹² . Concurrent with WebAgent. Self-hosted realistic websites for offline benchmarking; CUA scores 58.1% here ⁹ .
• OSWorld (arXiv:2404.07972, April 2024) ¹³ . Concurrent with V-IRL. The canonical benchmark for full computer use; CUA scores 38.1%, Computer Use 14.9–22.0% ^{7 9} .
• WebVoyager (arXiv:2401.13919, January 2024) ¹⁴ . Concurrent with V-IRL. Live-web agent benchmark using GPT-4V; CUA later matches human performance (87%) here ⁹ .

The technical relationship: WebAgent and V-IRL are method papers; Mind2Web, WebArena, OSWorld, WebVoyager are benchmark papers; Computer Use and CUA are production releases that report on the benchmark papers’ tasks. The four-paper cluster forms the evaluation infrastructure that the production releases are measured against.

[Reviewer Perspective] Strongest skeptical objection. The lineage’s two largest production releases (Computer Use, CUA) do not disclose their training recipes, training data, or hyperparameters. From a research-progress standpoint, this means the durable architectural lesson (screen + cursor as universal interface) can be borrowed, but the training-pipeline lesson cannot. Future open-source replication of these capabilities will have to re-invent the RL post-training recipe from scratch.

[Reviewer Perspective] Strongest paper-side rebuttal. Anthropic and OpenAI both face a tradeoff between transparency and competitive moat / safety. The system cards do disclose the prompt-injection defences, the safety classifiers, and the benchmark numbers; what’s withheld is the training recipe specifically. This is a defensible posture in a competitive frontier-LLM market where the training-pipeline IP is the moat.

What remains unsolved. Long-horizon reliability across many sequential steps. Even CUA at 38.1% OSWorld means more than half of multi-step computer-use tasks fail ⁹ . The compound-error model (MATH ENTRY 4) explains why; the architectural fix is not yet known.

Three future research directions (each grounded in a paper-specific gap).

1. Hierarchical agents with explicit error recovery. [Analysis] WebAgent’s planning failures account for 33–50% of errors on real websites ⁴ . A planner with explicit re-planning when the executor reports failure should reduce this; the architectural primitive is not yet standard.
2. Open replication of CUA-style RL post-training. [Analysis] The CUA training recipe is undisclosed ⁹ . An open replication that hits 50%+ OSWorld would be the durable contribution; this is plausibly achievable in 2026 given the open-source momentum on agent post-training (GRPO, AgentFlow).
3. Geographic / linguistic robustness for production computer-use agents. [Analysis] V-IRL surfaced cultural bias in vision encoders ⁶ ; the production Computer Use and CUA systems have not published equivalent regional breakdowns. A V-IRL-style geographic-robustness audit of CUA on global websites would close the loop.

12. Critical analysis

12.1 Strengths with concrete evidence

WebGPT. Established RLHF + browser-use as a viable production direction; 56% human-preference over demonstrators on ELI5 was the headline that gave RLHF credibility for tool-use settings ² .

WebAgent. Three-module decomposition delivered 50+ percentage-point gain over monolithic Flan-U-PaLM on real websites ⁴ ; HTML-T5’s local + global attention beat dense by 18 points on MiniWoB++ at the same training budget ⁴ .

V-IRL. Surfaced the cultural / linguistic bias of vision encoders systematically across 14 regions ⁶ ; the empirical finding is a durable contribution beyond the platform itself.

Computer Use. Productionised the screen-pixel interface as a developer API; the OSWorld number (22.0% with multi-step) is the public anchor against which subsequent releases are measured ⁷ .

CUA. Closed much of the gap to human performance on WebVoyager (87% vs. 100% oracle) ⁹ ; advanced OSWorld and WebArena state-of-the-art by 16–22 percentage points over Computer Use.

12.2 Weaknesses explicitly stated by the authors

WebGPT (per paper Section 6) ² . PPO can reward-hack the reward model; rejection sampling is more reliable in practice. The model can confidently cite plausible but incorrect references.

WebAgent (per paper Limitations) ⁴ . Modular approach increases latency and cost vs. single-LLM. Self-experience supervision was demonstrated on only 3 websites with <900 total episodes. Large 540B models are difficult to adapt based on code execution errors. Real-world evaluation cost beyond human supervision is unexplored.

V-IRL (per paper Limitations) ⁶ . Web-based movement restricted by incomplete street-view coverage. Feature matching prone to false deduplicates across viewpoints. Language-based agents lack sensory grounding without V-IRL. Computational costs for global-scale evaluation.

Computer Use (per Anthropic announcement) ⁷ . The capability is “experimental — at times cumbersome and error-prone.” Specific challenges include scrolling, dragging, zooming. Developers are encouraged to start with low-risk tasks.

CUA (per OpenAI launch) ⁹ . OCR challenges with complex strings (DNA sequences, API keys, cryptocurrency wallet addresses). The 38.1% OSWorld figure is explicitly framed as substantial progress, not as production-ready performance.

12.3 Weaknesses not stated by the authors (or understated)

[Reviewer Perspective] Computer Use, CUA training-recipe opacity. Neither announcement discloses the training data, reward signal, or RL recipe used. Replication is therefore impossible from public artefacts alone. This is a non-trivial loss to the research community.

[Reviewer Perspective] CUA benchmark-selection framing. The 87% WebVoyager headline matches “human performance,” but WebVoyager’s human baseline is itself a benchmark-design choice ¹⁴ . Different human-baseline definitions yield different headline numbers. Buyers should read the WebVoyager paper alongside the CUA claim ^{9 14} .

[Reviewer Perspective] WebAgent’s <900-episode real-world evaluation. Across the three production websites, the paper reports under 900 successful demonstrations used for finetuning ⁴ . The 65–80% completion numbers are on the same three websites; cross-domain generalisation is not demonstrated.

[Reviewer Perspective] V-IRL’s reliance on Google Maps Platform. The platform’s geographic coverage and cultural biases are bounded by Google’s data ⁶ . The “geographic bias” finding may itself reflect Google’s coverage bias rather than purely the vision models’ bias.

12.4 Reproducibility check

Paper	Code	Data	Hyperparameters	Compute	Weights	Eval set	Overall
WebGPT	Not released	Not released	Partial (appendix)	Not reported	Not released	ELI5 public	Partial
WebAgent	Not released (HTML-T5 weights unreleased)	Partial (Mind2Web public)	Partial (appendix)	Not reported	Not released	Mind2Web public; MiniWoB++ public; real-estate / social-media / map private	Partial
V-IRL	Released (project page virl-platform.github.io)	Google Maps API + 14-region task set	Partial	Not reported	N/A (off-the-shelf models)	Released	Reproducible
Computer Use	API access only	Not released	Not released	Not reported	API only	OSWorld public 13	Partially reproducible (results-only)
CUA / Operator	API access only	Not released	Not released	Not reported	API only	OSWorld, WebArena, WebVoyager public	Partially reproducible (results-only)

12.5 Methodology disclosure

Methodology — WebGPT

• Sample size: ELI5 evaluation; ~272 questions in the rated test set per the paper (specific number per Appendix) ² .
• Evaluation set: ELI5, TruthfulQA — both public.
• Baselines: Human demonstrators, top Reddit responses.
• Hardware / compute: Not reported in detail.

Methodology — WebAgent

• Sample size: 50 episodes per real-world website (real-estate, social-media, map); Mind2Web full test set; MiniWoB++ full task suite ⁴ .
• Evaluation set: Mind2Web public; MiniWoB++ public; three production websites private.
• Baselines: Single Flan-U-PaLM; MindAct; GPT-4; WebN-T5; Flan-T5-XXL.
• Hardware / compute: Not reported in detail.

Methodology — V-IRL

• Sample size: 14 districts across 12 cities; benchmark task counts per Section 4 ⁶ .
• Evaluation set: V-IRL Place Localisation, V-IRL Place Recognition + VQA, V-IRL VLN — all introduced in the paper.
• Baselines: Multiple off-the-shelf vision and multimodal models.
• Hardware / compute: Not reported in detail; bounded by Google Maps Platform quota.

Methodology — Computer Use, CUA

• Sample size: OSWorld (369 tasks per OSWorld paper) ¹³ ; WebArena (812 tasks) ¹² ; WebVoyager (~640 tasks) ¹⁴ .
• Evaluation set: All three public.
• Baselines: Prior state-of-the-art on each benchmark.
• Hardware / compute: Not reported; both products run on vendor cloud infrastructure.

12.6 Generalisability

[Analysis] The three-module decomposition (WebAgent) generalises across web domains; the empirical evidence is three websites. Screen-pixel interface (Computer Use, CUA) generalises across operating systems by construction; OSWorld covers Ubuntu, Windows, macOS ¹³ . V-IRL’s geographic-bias finding generalises by construction (cultural / linguistic diversity is the test signal), but the magnitudes are bounded by the 14-region sample.

12.7 Assumption audit

The shared assumption of sufficient observability is weakest for screen-pixel agents on text-heavy interfaces (OCR fails on complex strings per CUA’s own acknowledgement ⁹ ). The action-vocabulary completeness assumption is strongest for cursor + keyboard (covers any GUI by construction) and weakest for WebGPT’s six commands (cannot handle JavaScript-heavy modern sites). The reward-availability assumption is undisclosed for Computer Use and CUA, making the training pipeline opaque.

12.8 What would make the papers significantly stronger

[Analysis]

• WebGPT: A full system-card-style ablation of which training stage contributes most to the headline number.
• WebAgent: Cross-domain evaluation on a broader set of real-world websites; the 3-website evidence is the most fragile part of the empirical case.
• V-IRL: A finer-grained breakdown of the geographic-bias finding by language / culture / Google-Maps-coverage axes.
• Computer Use, CUA: Public release of training-recipe details and reward-signal specifications, or at minimum a system-card-style disclosure of the post-training pipeline.

13. What is reusable for a new study

REUSABLE COMPONENT 1: Three-module decomposition (plan, summarise, synthesise)

• What it is: WebAgent’s architectural split — one model plans next sub-instruction, one summarises the environment, one emits executable code or actions.
• Why worth reusing: 50+ percentage-point lift over monolithic prompting on real websites ⁴ .
• Preconditions: Access to two LLMs with different strengths (a smaller domain-expert planner, a larger code-emitting synthesiser). The pattern works at 3B + 540B as in WebAgent but should scale down (e.g., 7B + 70B).
• What would need to change: Domain-specific finetuning of the planner; the program-synthesis step may use a different language (Playwright instead of Selenium; pyautogui for desktop).
• Risks: Adds latency and cost (three model calls per step vs. one). Error compounding across three modules.
• Interaction effects: Works best when the summariser produces a focused enough snippet that the synthesiser can attend to it cleanly.

REUSABLE COMPONENT 2: Mixture-of-long-span denoising for HTML / structured-text pretraining

• What it is: Span-denoising pretraining with mean span length $\mu \in \{8, 64\}$ instead of standard $\mu = 3$.
• Why worth reusing: Captures semantic chunks rather than syntactic tokens in structured-text domains.
• Preconditions: A large structured-text corpus (HTML, XML, source code, log streams, configuration files). The recipe is corpus-agnostic; the span-length distribution is the load-bearing change.
• What would need to change: Span-length distribution may need re-tuning per domain (source code likely benefits from different $\mu$ than HTML).
• Risks: If $\mu$ is too large relative to input length, masking can swallow whole documents.
• Interaction effects: Pairs naturally with a long-context encoder (local + global attention or sliding-window).

REUSABLE COMPONENT 3: Screenshot + cursor + keyboard interface

• What it is: The universal action surface adopted by Computer Use and CUA — model observes a screenshot, emits cursor coordinates plus keyboard inputs.
• Why worth reusing: Works on any GUI by construction; no per-application API integration required.
• Preconditions: A vision-capable LLM that can reliably emit cursor coordinates given a screenshot.
• What would need to change: Action vocabulary may need extension (e.g., drag, multi-touch). Prompt-injection defences are mandatory.
• Risks: OCR failures on complex strings ⁹ ; small-element misclicks; prompt-injection from page content.
• Interaction effects: A separate prompt-injection screenshot classifier (per CUA ¹⁰ ) is now the production-default companion.

REUSABLE COMPONENT 4: Self-experience supervision pipeline

• What it is: Scripted agents generate trajectories; environmental feedback filters successful episodes; the agent is finetuned on filtered demonstrations.
• Why worth reusing: Lower demonstration cost than human-only data collection; the filter is automatic.
• Preconditions: A scripted baseline agent and an environmental success signal.
• What would need to change: The success signal is domain-specific; defining “execution error” cleanly is non-trivial outside web automation.
• Risks: The scripted agent’s distribution may be biased; finetuning on filtered self-experience can amplify these biases.

REUSABLE COMPONENT 5: Hybrid scoring for agent benchmarks

• What it is: [External comparison] Combines deterministic checks (where evidence is unambiguous) with structured LLM judging (only for semantic dimensions). Adopted by recent benchmarks like Claw-Eval-Live.
• Why worth reusing: Reduces LLM-judge variance while preserving the ability to score semantic outputs.
• Preconditions: Each task has a rubric splitting deterministic from semantic dimensions.
• What would need to change: The rubric design is domain-specific.

Dependency map. Component 1 (three-module decomposition) depends on Components 2 (specialised encoder) and the program-synthesis tradition. Component 3 (screenshot interface) is independent of Components 1–2 and is the current production default. Component 4 (self-experience supervision) is independent of all others and is a training-pipeline contribution. Component 5 (hybrid scoring) is an evaluation primitive that all of Components 1–4 should be measured against.

Recommendation. [Analysis] For a team building a new agent product in 2026, the highest-value reusable components are (a) the screenshot interface (Component 3) for any UI-agnostic workflow and (b) the three-module decomposition (Component 1) for the planner-executor split. The HTML-T5 recipe (Component 2) is worth borrowing for any vertical with heavy structured text.

[Analysis] The type of new study that benefits most: an open replication of CUA-style RL post-training on top of an open-weights base model, evaluated on OSWorld and WebVoyager, with the training recipe disclosed.

14. Known limitations and open problems

14.1 Author-stated limitations

Per Section 12.2. WebGPT: reward hacking. WebAgent: latency, narrow real-world evidence, hard to adapt 540B to errors. V-IRL: street-view coverage gaps, feature-matching noise, ungrounded LLM weakness. Computer Use: scroll / drag / zoom challenges, low absolute OSWorld score. CUA: OCR on complex strings, 38.1% OSWorld below production-ready threshold.

14.2 Limitations beyond the authors’ framing

[Analysis] [Reviewer Perspective]

• Training-recipe opacity for Computer Use and CUA. Neither vendor discloses training data, reward signal, or RL recipe. Independent replication is impossible from public artefacts ^{7 9} .
• Compound-error fragility. Per MATH ENTRY 4, every paper in the lineage faces the same multiplicative reliability problem. None proposes an explicit architectural fix.
• Cross-paper benchmark partition. Mind2Web, WebArena, WebVoyager, OSWorld each measure something different; aggregate cross-benchmark comparisons are unsound.
• Geographic / cultural robustness. V-IRL surfaced the issue for vision encoders ⁶ ; no production computer-use agent has published equivalent regional breakdowns.

14.3 Technical root causes

Compound errors are the dominant force. Long horizons amplify any single-step unreliability; 94% step reliability becomes ~38% task reliability over 15 steps. The architectural answer (explicit re-planning, hierarchical task structure, human-in-the-loop checkpoints) is not yet standard.

14.4 Open problems

1. Long-horizon reliability above 70% on OSWorld. Currently 38.1% ⁹ . Crossing 70% would mark the transition to viable hands-off automation for most workflows.
2. Open replication of CUA-style post-training. Required for any open-source ecosystem to match closed frontier agent capability.
3. Generalisable prompt-injection defences. CUA’s screenshot classifier is a production answer ¹⁰ ; the general theory of computer-use-agent attack surfaces is undeveloped.
4. Geographic and linguistic robustness in production. A V-IRL-style audit of Computer Use and CUA on global tasks would close the loop V-IRL opened.

14.5 What a follow-up paper would need to solve

[Analysis] To advance the lineage materially in 2026–2027, a follow-up paper would need to (a) disclose a reproducible RL post-training recipe that hits 50%+ OSWorld on an open-weights base, (b) demonstrate explicit error-recovery architecture that reduces compound-error fragility, and (c) publish a geographic / linguistic robustness audit comparable to V-IRL’s. Combining (a)–(c) in a single open-source release would be the strongest single contribution the lineage could receive in the next 18 months.

How this article reads at three depths

For the curious high-school reader. Between 2021 and 2025, AI agents that use web browsers and computers got dramatically better at completing real-world tasks. WebGPT (2021) wired a language model to a simple text browser. WebAgent (2023) broke the job into three smaller jobs done by three different models. Computer Use (Anthropic, 2024) and Operator (OpenAI, 2025) showed the model a screenshot and let it move a cursor — like a person does. The newest agent matches human performance on live websites 87% of the time but still fails more than half of harder full-computer tasks. The pattern is interesting: instead of teaching the model to read the page’s underlying code, designers gave up and let it look at the screen the way humans do — and that worked better.

For the working developer or ML engineer. The screen-as-interface convergence is the durable architectural insight. If you’re building an agent product in 2026, default to a screenshot-plus-cursor loop with HTML-DOM as an optional structured side-channel rather than the primary interface. Add a planner-executor split (WebAgent-style three-module decomposition gives 50+ percentage points over monolithic prompting on real websites) and a prompt-injection screenshot classifier (CUA-style). Treat web-only workflows as near-term production-ready (87% WebVoyager) and full-desktop automation as the medium-term frontier (38.1% OSWorld). Avoid betting the stack on a single benchmark number — the partition between MiniWoB++ / Mind2Web / WebArena / WebVoyager / OSWorld is meaningful and a strong score on one does not transfer.

For the ML researcher. The most enduring contribution in the lineage is WebAgent’s three-module decomposition; the most consequential opacity is Computer Use / CUA’s undisclosed RL post-training recipe. Mixture-of-long-span denoising ($\mu \in \{8, 64\}$) for HTML transfers to any structured-text vertical and is worth borrowing for code, XML, log-stream pretraining. The compound-error model explains why benchmark numbers in the 30–60% range are consistent with 90%+ step reliability; the architectural fix for compound-error fragility remains open. The strongest objection to the lineage’s headline numbers is benchmark-selection framing — the 87% WebVoyager claim should be read against WebVoyager’s own human-baseline definition, not as a universal “human-level agent” claim. The strongest open contribution a 2026–2027 paper could make would be a reproducible open-source RL post-training recipe matching CUA on OSWorld and WebVoyager.