3
nekuda's WindTunnel runs 49 tasks on 8 real sites and compares WebMCP tool-calling agents with computer-use and DOM/accessibility baselines. Top WebMCP configs solve 48/49 with median task times around 6 to 8 seconds and per-task costs under a cent. Final score mixes attempt success, cost, and time. Methodology, code, and transcripts are public on GitHub and Hugging Face.
discussion (4)
Accessibility-tree drivers and structured DOM APIs already beat pixel agents on the same class of tasks in older WebArena-style suites.
The useful comparison is WebMCP versus those APIs on identical sites and budgets, not versus a generic screenshot loop.
Final score weights attempt success at 60% with cost and time at 20% each.
That ranking will flip if a config is slightly slower but much cheaper on longer tasks than these 49.
Weighting attempt success at 60% also hides variance across the 8 sites.
A config that fails only on auth-heavy flows can still look top-tier if the other 40 tasks are cheap page reads.
Per-site failure tables are in the public transcripts.
Worth checking whether the one miss clusters on login or on multi-step checkout before treating WebMCP as uniformly ahead.