Some background first. A web agent is a system that lets a large language model drive a browser on its own. It looks at the page (a screenshot or the DOM), decides where to click and what to type, and works step by step through tasks like "forward this email to my boss" or "place a stop-loss order at this price." Benchmarks for these agents usually work the same way: prepare a set of websites and tasks, let the agent attempt them, and count how many it gets right.
Say a web agent scores 60 on a benchmark. What does that number actually tell you? Did it fail to understand the page, lose track of where it was, or get fooled by a fake "Saved" message? Most web benchmarks can't answer that, because all they report is a single total score.
BreakingWeb (paper: Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions) is built to answer exactly that question. The idea is simple. First, build seven complete websites so that every task can be graded precisely from backend state. Then run each task a second time in a "broken" version: a request times out, a save pretends to succeed, a look-alike decoy is planted in the data. The gap between the two scores measures the agent's weakness in one specific ability.
The result: across six frontier agents, pass rates dropped by an average of 22.9 points on the intervention versions, while humans dropped only 5.7 to 10 points. Agents mostly don't fail because they can't click. They fail because they believe they're already done.
This post records how the benchmark was designed. It covers six things:
- Why the environments started small and grew step by step
- The "live" mechanics inside the environments, such as stock prices that move on their own
- How every task was made verifiable
- How many people wrote tasks to one shared standard
- Scoring and matching, and how to visualize them
- What the results show
1. Start small, then scale up
The project didn't begin with seven websites. It started with something very small, and each time we found "this is too shallow, it can't tell models apart," we pushed it one step bigger. Here are those steps in order.
October 2025. The earliest version lived in a collaborator's repository, and the web pages weren't real servers at all. They were "simulated" by an LLM: the agent clicked a button, and the LLM generated what the next page should look like. The upside was that any website could be "built." The downside was that there was no real backend state. If the agent said "I archived the email," there was nowhere we could check whether that actually happened.
February 12, 2026. I rewrote it as a standalone benchmark, which eventually settled into ten single-page HTML challenges. Each page tested one thing: a setting buried deep in a menu, a number that only matched up after flipping through two pages, a form that looked submitted but wasn't. The pages were real now, and correctness could be checked from page state.
Early March 2026. The single-page version hit a ceiling quickly. The pages were too shallow. Stronger agents saw through them immediately, and weaker ones got stuck on things unrelated to ability, like a button written in an odd way. The more fundamental problem was that the pages shared no state. We couldn't test "remember what you've done," and we couldn't test "go back and check after you finish," and those two were exactly what we most wanted to measure.
March 13, 2026. I wrote a design doc whose title was simply the pivot to complex environments. The core idea: instead of building many shallow pages, build a few complete websites, with a real React frontend and a real database backend, and judge each task from backend state.
March 19, 2026. The first complete environment, Gmail, went live with 20 tasks. It was the testbed: inbox, threads, labels, search, settings. Every later environment copied its structure. At the end of the month, every task got its first "broken" version.
April 1 to 10, 2026. This was the real burst. Robinhood (April 1, with the price engine written the same day), Amazon and Reddit (April 8), Booking (April 9), LMS and Patient Portal (April 10): six new environments landed in ten days. On April 4 we hooked up the Browser-Use harness and ran the first full sweep on Gmail: 65% pass rate on the clean version, 48% on the intervention version. It was the first time the gap showed up clearly in numbers, and it convinced us the direction was right.
After April 16, 2026. The environments were big enough, and the new problem became how to judge correctness. Each task had been written as a set of hand-coded checks, and they kept drifting apart. So we rewrote all of scoring around a canonical diff (more in Section 4), and 157 commits migrated the tasks of all seven environments over one by one. The repo got 627 commits that month, about 60% of the whole project.
May to early June 2026. Adding human baselines, auditing trajectories, and fixing tasks round after round. The last commit was on June 3.
Today each environment is a React frontend plus a FastAPI backend, and all of the data is synthetic. Across the seven environments there are 337 API endpoints and 519 tasks, with roughly 35,000 lines of frontend TypeScript.
Every time we scaled up, it was because the previous version couldn't separate models. That is the most direct lesson I took from the process: the depth of the environment is itself part of the measuring instrument. When pages are shallow, strong and weak models both get them right. Differences in ability only show up when a task requires going three menus deep, cross-checking two pages, and remembering a number seen five minutes earlier. By the seven-website version, the strongest and weakest text agents were already nearly 40 points apart on clean tasks (72.1% vs. 33.1%).
2. Making the environment live: Robinhood's price engine
Static websites have a basic problem: the world the agent sees on its first look is identical to the one it sees on its last. Reality isn't like that. Prices move, orders fill while you aren't looking, alerts fire out of nowhere. What I wanted to test was precisely whether an agent realizes "the world has changed, I need to look again."
So the Robinhood environment has a PriceEngine. It rests on three design choices.
First, it advances on reads, with no background thread. Every API request first calls state.tick(), which uses wall-clock time to work out how many ticks have elapsed (one every 2 seconds by default) and advances prices to that tick. When nobody is making requests, the world stands still.
Second, prices follow a script, not a random walk. Each stock has a set of keyframes [tick, target price], linearly interpolated, with a layer of ±0.3% noise on top. The noise is pregenerated from a fixed seed, so the same task produces exactly the same price path on every run, and results are reproducible.
kf = trajectory.keyframes
if tick <= kf[0][0]:
base = kf[0][1]
elif tick >= kf[-1][0]:
base = kf[-1][1]
else:
for i in range(len(kf) - 1):
t0, p0 = kf[i]
t1, p1 = kf[i + 1]
if t0 <= tick <= t1:
progress = (tick - t0) / (t1 - t0) if t1 != t0 else 0
base = p0 + (p1 - p0) * progress
break
if trajectory.noise_pct > 0 and tick > 0:
noise_val = noise_list[tick % len(noise_list)]
base *= 1 + noise_val * (trajectory.noise_pct / 100)Third, when a price moves, the consequences cascade. cascade_update does five things in order: update the stock price and bid/ask; update profit and loss on holdings; check open orders and fill any limit or stop orders that should execute, creating the position change, transaction record and notification along the way; check price alerts; and finally recompute total account value. A validator also rejects unreasonable scripts, such as a single tick moving more than 2%.
A concrete task: rh_live_stop_loss_execution. The instruction is "You hold 5 shares of NVDA. Set a stop-loss order 8% below the current price." In the script, NVDA falls from 875 to 800 and crosses the stop line after about 50 seconds. The agent has to compute the right price, place the right order, and then wait. The correct end state isn't just "an order was placed." It's that the order filled, a sell transaction exists, a notification exists, and the NVDA position is gone.
This also makes grading harder. An agent that does the task right leaves a chain of related changes in the backend, and the grader has to know which of those changes are supposed to be there. Section 4 comes back to this.
3. Breaking the web: four layers of interventions
Once we had clean tasks that could be graded exactly, the next step was to give every task a "broken" version. Interventions come in four layers, ordered by when they take effect:
My favorite is misleading_success. The agent clicks "Save," and for the first few requests nothing reaches the real backend. A middle layer simply returns HTTP 200 with {"success": true, "message": "Saved."}. A green "Saved" toast appears on the page, but nothing is written to the database. Human users occasionally get caught by this too, and a good agent should go back and confirm the setting actually took effect.
{"success": true, "message": "Saved."}
The current 535 intervention variants cover all 519 tasks. The most common techniques are adding decoys (236 times), silent failures (179), erroring first and then succeeding (71), returning stale data (70), and faking success (52). Each variant declares which cognitive ability it targets, out of seven: grounding, state tracking, backtracking, verification, planning, exploration, and patience.
Under the paper's classification, these techniques fall into 29 intervention families (the variant files contain 52 distinct layer-plus-action combinations).
Clean and intervention versions come in pairs, and that is the core of the whole design. Same agent, same task, one intervention apart: the drop in score can be attributed to that ability.
4. Making every task verifiable
The early checks were hand-written expressions, one per line, and they had plenty of problems. A real example: a Patient Portal task asked the agent to book one shot for each overdue vaccine, and the check was written as len(new_appointments) >= 1. An agent booked two unrelated appointments at random and got full marks.
In mid-April we rewrote all of grading around a canonical diff. The core idea fits in one sentence: whether a task is done correctly is determined entirely by the change from initial state to final state. The author writes down the changes that should happen, and the changes the agent actually caused are compared against them.
A task's YAML looks like this (with a few invariants removed):
task_id: gmail_star_email
instruction_template: Find the email from Alice Chen with subject "Project Update
— Q1 Milestones" and star it.
difficulty: easy
canonical_diff:
update:
- entity: emails
where: {id: {expr: "x == target['target_email_id']"}}
changes: {is_starred: {eq: true}, labels: {superset: [starred]}}
invariant:
- collection: state.emails
filter: "a.id != target['target_email_id']"
preserve: ALL
- collection: state.sent
preserve: ALL
named_invariants:
- {name: Agent starred the target email, ref: update[0], severity: high}
- {name: Agent did not send emails, ref: invariant[1], severity: high}It compiles into two kinds of checks:
- Positive obligations: every entry under
create,updateanddeleteis a change that must happen. - Negative invariants: collections under
invariantmust not be touched, andconstraintsare additional boolean conditions.
- emails[target].is_starred == true
- emails[target].labels ⊇ {starred}
- emails where id ≠ target: unchanged
- sent: unchangedAgent did not send emails · high
The most useful thinking tool for writing these YAML files is the three-bucket model. Every collection in the backend must go into exactly one bucket:
- Must change: where the changes the task requires happen.
- Frozen:
preserve: ALL, nothing may change. - Ignored: irrelevant changes will happen here, and the spec explicitly declares they don't matter.
If a collection isn't placed in any bucket, any change to it is treated as an unexpected side effect and penalized at high severity. This rule forces authors to think the backend through. When you cancel a Booking reservation, does the system automatically create a rebooking suggestion? When a Robinhood limit order fills, does a notification get added? All of these knock-on changes have to be written down in advance.
5. Many authors, one standard
The 519 tasks weren't written by one person. The paper has 11 authors, the repo has commits from about five people, and I wrote roughly 70% of them. With more people, the biggest risk is that everyone has a different idea of what a good task is.
Our approach was to write the standard down as a document, then turn the document into tests.
TASK_GENERATION_STANDARD.md set a few hard rules:
- The instruction must state the action, the target, the selection criteria, the completion criteria, the exclusions, and the tie-breaking rule.
- Vague words like "appropriate," "reasonable" and "as needed" are banned. If you write "latest," you have to say which field is compared.
- The primary evaluation must be a state change. It can't check whether a keyword appears in the agent's reply. No exceptions.
- Every field the instruction asks the agent to read or compare must actually be rendered on the page, to prevent information asymmetry.
- Harder tasks require more checks: easy needs 2 positive and 1 negative, frontier needs 6 positive and 3 negative.
The canonical diff era added a 1,285-line authoring protocol that splits writing a task into ten steps: parse the instruction, list backend side effects, identify entities, classify each field, assign buckets, derive the positive diff, derive invariants, name them, check feasibility, and preview plus adversarial testing. "No step is optional."
Before entering the dataset, every task goes through a six-stage pipeline:
- 1Has a canonical_diffpass:
- 2Schema loadspass:
- 3Preview renderspass:
- 4Round-trip smoke testpass:
- 5Adversarial batteryfail:rejected
- 6Equivalence vs past trajectoriesnot run
- predicate
- create appointments, count ≥ 1
- wrong state
- one unrelated appointment, no vaccine booked
- score
- 1.00, should be below 1
Stage 5 is the most interesting one. For every condition the author wrote, adversarial testing automatically synthesizes a wrong end state that violates it. If that wrong end state still gets full marks, a check is missing, and the task is sent back.
Writing down a standard doesn't mean there are no bugs. At the end of April we ran all 1,049 runs with Sonnet 4.6, then turned around and audited the grader. We clustered runs by the description of the failing check and decided, one cluster at a time, whether it was a grading bug, a task design bug, a frontend bug, or a genuine agent failure. The audit turned up more than 30 failure patterns, and most of them have been fixed.
A few typical ones:
- Seed data took the first match globally, but the instruction said "in this course," so more than ten tasks scored 0 on the clean version.
- Canceling a reservation automatically created a rebooking suggestion, nobody had whitelisted that collection, and every run lost 0.2.
- The operation the task asked for had no entry point anywhere in the UI.
One very plain diagnostic rule helped a lot: if the clean version has a high failure rate, suspect the task before the model. It made me realize that the hardest part of building a verifiable benchmark is verifying your verifier.
6. Scoring, matching, and visualizing it
Scoring works like this:
- Diff the initial snapshot against the final backend state to get every change the agent caused.
- For each positive obligation, look for a matching change.
- Check invariants and constraints.
- Run one "side-effect sweep" over all the changes nobody claimed.
The hard part is the one-to-many case in step 2. Take "book one shot for each overdue vaccine." On the author's side there is a set of slots (one per vaccine). On the agent's side there is a set of observed changes (the appointments it actually booked). If an appointment satisfies a slot's conditions, an edge connects them. Then we compute the maximum matching of this bipartite graph.
We didn't use a library. _bipartite_matching is a hand-written augmenting-path algorithm (Kuhn's). The fraction of slots matched is the fraction of credit earned. Extra appointments are left for the side-effect sweep to penalize, so booking a pile of them and hoping for the best doesn't pay off.
The final score is:
Penalties scale with severity: critical 0.30, high 0.20, medium 0.15, low 0.10. Separately, a task only counts as passed if nothing failed, so a run can score 0.8 and still not pass.
One more detail. If a change hits the right entity but has the wrong attributes (say, the right vaccine on the wrong date), it counts once as a missing positive obligation and isn't penalized a second time as an unexpected side effect. Before the audit, this double penalty affected several categories of tasks.
One booking per overdue vaccine.
- Tdappass:
- HPV dose 2pass:
- Flupass:
- Tdap · Dec 2
- HPV · Dec 3
- Flu · Dec 4
This figure actually comes from our debugging panel. The evaluation sidebar in control.html draws every one-to-many entry as three columns: the required slots, the matching, and the changes the agent actually produced. Solid lines are the chosen matching, dashed lines are other possible edges. When writing tasks or fixing bugs, one glance at this view is much faster than reading a screen of logs.
7. What the results show
The paper evaluates six text-mode agents (using the accessibility tree), three pixel-mode agents that see only screenshots, and humans. Each agent runs every one of the 519 tasks once in the clean version and once in the intervention version:
(Clean scores are the intervention scores from Table 2 of the paper plus the drop.)
A few things stand out.
The drop is large, and it's everywhere. The six text agents drop 22.9 points on average. Of the tasks each agent solves in the clean version, 39.6% to 68.0% fail under intervention.
Humans are barely affected. Once familiar with the tasks, humans drop only 5.7 points. On runs both completed, humans are also about five times faster than agents (median 34 seconds vs. 184 seconds for Gemini-3.1-Pro). The abilities with the widest human-agent gap are patience, verification and backtracking, where the difference in drop is 52.4, 38.3 and 33.3 points respectively.
The paired design really does break the total score apart. Take Gemini-3.1-Pro. It loses 39.4 points on verification, 37.8 on backtracking and 34.5 on grounding, but only 9.6 on planning. A "72-point agent" becomes a very specific list of weaknesses. Exploration has the smallest drop, but only because its clean baseline is the lowest to begin with, so it shouldn't be read as a strength.
Agents mostly fail by believing they're done. Among 3,037 classified failed runs, the largest category is accepting a fake success (1,454), followed by retry loops (790) and declaring completion too early (433). For text agents, 75% of failures are belief failures of this kind: the page says "Saved," and the agent believes it. Screenshot-only agents are the opposite. 57% of their failures are action failures, where they can't even click the button accurately.
fail:75%of text-agent failures are belief failuresThat is why the fake "Saved" toast from Section 3 is my favorite intervention. It hits today's agents exactly where they are weakest.
8. What works, and what isn't done yet
What I think this design gets right:
- Paired comparison. One total score becomes a table broken down by ability. The same agent might drop a lot on backtracking and barely at all on planning. That is far more useful than "it scored 40."
- Grading from backend state. A fake "Saved" on top of a write that never happened counts as a failure. The UI can fool the agent, but it can't fool the grader.
- Reproducibility. Self-hosted, synthetic data, fixed seeds, scripted price paths. Every sweep records its git SHA.
- Two harnesses. Text mode uses the accessibility tree and pixel mode sees only screenshots, which separates model ability from harness effects.
- A human reference. 140 tasks, each in clean and intervention form, with every participant doing one cold attempt and one familiarized attempt, for 630 human attempts in total.
- Auditing the benchmark itself. Adversarial tests, solvability tests and trajectory audits are the infrastructure that makes the scores trustworthy.
What isn't done yet:
- All seven environments are English-language consumer websites, and the agents tested are mostly closed frontier models.
- The synthetic environments have none of the rate limiting, fraud controls or third-party scripts of real websites, so whether results transfer to the real web is still an open question.
- The clean baseline for exploration is low, so that column can't be read as a strength yet.
- Robinhood's prices advance on wall-clock time, so a slow agent can miss a price window. That mixes speed into the patience measurement.
If I had to sum up the project in one sentence: the environment has to be deep enough to separate abilities, the grading has to be strict enough for the score to mean something, and making the grader that strict takes more work than building the environments.