Book a 30-min call
← cd ../blogs
$ cat posts/browser-agents-the-honest-ceiling.mdx

Browser agents: the honest ceiling on unattended web automation

May 5, 2026 · ImmovableTech Team

  • Agentic AI
  • Evaluation
  • Production AI

The demo is not lying to you

When a browser agent books the flight on the first try in front of a room, that recording is usually real. When the same agent goes into a production workflow and completes a third of its runs, that is also real. Both facts come from the same system, and the thing that reconciles them is not model quality. It is arithmetic.

A task that takes twenty agent-controlled steps, where each step succeeds independently 95% of the time, finishes 35.8% of the time. Push per-step reliability to 99% and the same twenty-step flow finishes 81.8%. Drop to 90% and you are at 12.2%. The demo is one sample from a distribution with a long tail of failure, and it was selected for the retelling.

That is the whole argument, and everything below is the detail that makes it actionable: what the benchmarks actually measure, why steps fail in ways a better model will not fix, and why the engineering response is to have fewer agent-controlled steps rather than better ones.

The arithmetic, and where it is wrong

The compounding maths above assumes step failures are independent, and they are not. Failures cluster: a stale DOM snapshot causes three consecutive mis-clicks, a rate limiter poisons the rest of the session, an agent that has mis-read one page tends to mis-read the next. Positive correlation between failure events means fewer runs fail than the naive product suggests, so treat 35.8% as a pessimistic floor rather than a forecast.

The shape survives the correction, and the shape is what matters. Success rate decays geometrically in trajectory length, so the variable worth attacking is not the per-step rate but the exponent. Halving the number of agent-controlled steps does more for end-to-end reliability than any prompt you will write this quarter. A twenty-step flow at 95% finishes 35.8% of the time; a five-step flow at the same per-step rate finishes 77.4%.

This is also why step budgets are load-bearing rather than a cost control. WebBench capped agents at 50 steps per run. At 95% per-step reliability, a trajectory that genuinely needs 50 steps completes 7.7% of the time, which means a task that long is not a reliability problem you can tune — it is a task that should not have been given to an agent in that shape.

What the benchmarks report, and what the headline hides

WebBench, built by Halluminate with Skyvern, is the one we keep returning to because it separates the two things most benchmarks average together. It covers 5,750 tasks across 452 live websites sampled from the global top 1,000 by traffic, of which 2,454 are open-sourced. Crucially it splits READ tasks — navigate and extract — from state-changing ones: create, update, delete and file manipulation. Success is not a string match; every trajectory was labelled by a human annotator watching a screen recording, with an explicit “bad task” label to drop tasks that broke when the site changed.

On results dated 30 May 2025, the best fully automated agent scored 66.0% overall. That is the number that gets quoted. The split underneath it is the number that matters: five of the seven tested agents cleared 70% on READ tasks, while the best fully automated agent on non-READ tasks managed 46.6%.

The aggregate is misleading by construction, and it is worth being precise about why. READ tasks are 1,580 of the 2,454 open-sourced tasks — 64.4% of the set. An overall score is therefore dominated by the easy half of the benchmark. If your use case is reading a portal, the headline is roughly honest. If your use case is submitting something, you should be reading the 46.6%.

One detail in the methodology sharpens this further. WebBench handed agents their login credentials directly in the prompt, along with an instruction to stay on one domain. The authentication problem was partly solved for the agent, and the state-changing score was still under 50%.

Removing the infrastructure excuse

The obvious rebuttal to live-web benchmarks is that they measure the internet’s hostility to automation rather than the agent — proxies blocked, CAPTCHAs unsolved, Google Auth flagging a fresh browser profile as a bot. WebBench’s own error analysis separates exactly those infrastructure failures from agent failures, and grants they are a meaningful share.

REAL, published at the NeurIPS 2025 Datasets and Benchmarks track, is the experiment that removes the excuse. It runs 112 tasks against deterministic high-fidelity replicas of 11 real sites, with locked data, fixed date ranges, no anti-bot layer and sessions that start already logged in. No proxy failures, no CAPTCHAs, no flaky network, perfect replayability. Scoring is a binary outcome reward: programmatic checks of website state for action tasks, rubric-guided LLM judging for retrieval.

The best model in the paper reached 41.07%. A human baseline on the same tasks, given a strict three-minute limit and no prior knowledge of the sites, reached 96.42%.

Online-Mind2Web puts a third methodology on the same question and lands in the same place. Its authors titled the paper “An Illusion of Progress?” and argued directly that reported web-agent results were over-optimistic; their benchmark is 300 realistic tasks across 136 live websites, graded by an LLM judge calibrated to roughly 85% agreement with human raters. On Princeton’s HAL leaderboard, which independently reproduces the runs it lists, the best verified result is 42.33%.

Three benchmarks, three incompatible methodologies — deterministic replicas with programmatic state checks, live web with human annotation, live web with a calibrated judge — converging on roughly 41% to 47% for tasks that change state. That convergence is the single most useful fact in this post. It is not one lab’s harness being unfair.

Two caveats we would want a reader to hold. These figures are agent-plus-scaffold-plus-infrastructure results, not model scores; the same model posts different numbers on different harnesses. And they are 2025 figures. WebBench’s verified leaderboard entries are still dated 30 May 2025 and Princeton’s HAL team have posted that they have paused adding new models. We would rather cite a stale number we can trace to a paper than a current one we cannot.

Why the steps fail

Four failure classes account for most of what we see, and none of them is fixed by a smarter model.

Authentication and bot detection. This fails before the agent reasons at all. Datacentre IP ranges get blocked, CAPTCHAs are specifically engineered to resist browser automation, and OAuth providers score a session on device fingerprint and timing that a fresh automated profile fails. The agent never sees the page.

Dynamic DOM and late-loading content. The agent plans against a snapshot that has already expired. Cookie banners, subscription modals and consent overlays appear mid-session with no instruction covering them, and the run stalls at the obstruction. WebBench’s annotators logged “cannot close a popup dialog” as a recurring, distinct failure mode.

Non-idempotent actions. This is the one that turns a reliability problem into an incident. Reading a page twice is free. Submitting a form twice creates two records. The standard engineering reflex — retry on failure — is unsafe the moment a step has side effects, and the agent frequently cannot tell whether its previous attempt landed. REAL’s authors describe agents that add one of two items to a cart, misjudge their own state and proceed through checkout anyway. The failure is not the mis-click; it is the confident commit afterwards.

No rollback for a half-completed form. A five-page application with server-side state at each step has no transaction boundary. When the agent gives up on page four there is no ROLLBACK, and often no UI affordance to undo. You are left with a partial record in someone else’s system that your code did not create and cannot clean up.

Notice that three of these four are properties of the web, not of the model. They will still be there when the next frontier model ships.

Shrink the exponent

The design rule we have converged on: give the agent the decisions and give deterministic code the navigation.

Most real workflows are not twenty ambiguous steps. They are eighteen mechanical steps and two genuine judgement calls — which of these invoices matches the reference, is this the right delivery address, does this exception need a human. Those two are what a model is actually good at. The other eighteen are page.goto and a selector, and a Playwright script executes them at effectively 100% per-step reliability with a stack trace when it does not.

from playwright.sync_api import sync_playwright

# Deterministic shell owns navigation; the model owns one decision.
with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto('https://portal.example.com/invoices')
    page.get_by_role('button', name='Filter').click()

    rows = page.get_by_role('row').all_text_contents()
    target = choose_invoice(rows)  # the only model call in this flow

    page.get_by_role('link', name=target).click()
    page.get_by_role('button', name='Download PDF').click()

That collapses a twenty-step agent trajectory to a one-step one. The compounding term disappears, and what is left is a deterministic script whose failures are ordinary software failures: a selector breaks, you get an exception, you fix the selector.

The trade is real and we should name it. You lose generality. The agent that drives the whole browser handles a portal redesign that a hardcoded selector does not, and maintaining selectors across a fleet of vendor portals is genuinely tedious work. Our position is that this trade is worth making for any flow that changes state, and often not worth making for read-only extraction, where the benchmarks say agents are already reasonable. Read broadly, write through a narrow deterministic gate.

Where you cannot shrink the trajectory, make the remaining steps recoverable. Idempotency keys on anything that submits. An explicit verification step after every state change that reads the resulting state back rather than trusting the action’s return. Checkpoints that let a run resume rather than restart, because restarting a partially completed non-idempotent flow is how you get duplicates. This is the same discipline that made our MCP tool schemas work: constrain what the model can do, then verify what it did.

What we’d do differently

We spent about three weeks trying to raise a portal-automation agent’s per-step reliability with better prompting, richer accessibility-tree context and a model upgrade. We got real improvement — the kind that looks good on a chart — and it barely moved end-to-end completion, because the flow was long enough that the exponent dominated. We should have measured trajectory length on day one and rewritten the navigation before touching the prompt. That would have been an afternoon of arithmetic instead of three weeks.

The more expensive lesson was about retries. Our first version retried failed steps with a generic backoff, which is correct for reads and wrong for writes. We produced duplicate submissions in a staging environment before we understood that “retry the step” is not a safe default when the step has side effects, and that an agent’s own report of whether an action succeeded is not evidence. It was staging, so it cost us cleanup rather than a customer. It should not have taken a duplicate record to teach us something the REAL paper documents as a standard failure mode.

We also over-trusted our own evaluation early on. We scored runs on whether the agent reported success, which is precisely the thing these agents are bad at. Every benchmark we now take seriously — WebBench with human annotators, REAL with programmatic state checks — grades the resulting state, not the agent’s account of it. Our internal evals do the same now.

References

We design and evaluate browser and computer-use agents as part of our AI & Machine Learning Engineering practice. Talk to us if you have an agent that demos well and will not hold up unattended.