BreakingWeb: Finding Where Web Agents Break

Most web-agent benchmarks report a success rate. That tells you an agent is weak, but not whether it misread the page, lost track of state, or never checked its own work. BreakingWeb answers that question. Every task has a clean version and an intervention version that changes exactly one thing. The drop between the two isolates a single capability: grounding, planning, state tracking, backtracking or verification. Key contributions: - 519 matched task pairs across 7 self-hosted React/FastAPI environments and 29 intervention families - A canonical-diff evaluator: task specs compile to positive obligations and negative invariants over backend state, graded with bipartite maximum matching between target slots and observed mutations. Grading is deterministic and immune to spoofed UI state. - A four-layer deterministic fault-injection stack (seed, server, network, DOM) - Playwright harnesses for accessibility-tree (Browser-Use) and screenshot-only (BrowserGym) agents - Evaluation of 9 frontier agents against a 140-task human panel