Modern Web Guidance Evals

Measurably better web code with Modern Web Guidance

Nightly A/B runs give Claude Code, Codex CLI, and Antigravity CLI identical tasks with and without Modern Web Guidance, grading output in a real browser.

  • Best-practice pass rate

    …

    pass-rate lift

    Loading…

  • Automated accessibility checks

    …

    Not a full WCAG audit

    Loading…

Latest nightly runs · loading…

Inspect a paired run

Side-by-side browser checks from one nightly run per agent (not an aggregate).

Task prompt

Loading task prompt…

By agent family

Latest nightly suite per agent. Model and per-guide accessibility breakdowns below.

Agent Pass rate Accessibility Sample
Loading…———
Breakdowns by model and guide
days
How we test & measure
Controlled A/B tasks
Each task pairs a starter web app with a feature prompt (for example, "add search-term highlighting"). Every agent, model, and prompt runs twice: once without MWG and once with MWG installed as a skill.
Grading & pass rate
Playwright opens the finished app in a real browser and checks the DOM, styles, and source against each requirement. Pass rate is checks passed divided by checks run (runs that crash or time out before grading are left out, along with their paired run). Lift is shown in percentage points (pp): 25% → 90% is +65 pp.
Do graders penalize valid alternative code in "Without MWG"?
Graders assert rendered browser behavior and spec-compliant accessibility semantics (such as computed roles, focus movement, and keyboard interaction) rather than guide-specific variable names or markup structure. When an audit finds a grader that rejects a valid alternative implementation, we broaden the assertion in the open-source repository and re-score the suite.
Why isn't "With MWG" 100%?
Task prompts never mention MWG, so the agent has to recognize when a guide applies and follow every detail. Misses come from skipped steps (such as an older-browser fallback), missed guide lookups (see Search Quality), or strict grader edge cases.
Automated accessibility checks
Pass rate across automated browser assertions verifying foundational platform mechanics across five dimensions: keyboard support, focus and inert, prefers-reduced-motion and contrast, ARIA roles and states, and form labels. Automated checks catch common AI omissions in code but only cover a subset of accessibility requirements—they do not measure full WCAG compliance or replace manual and assistive-technology testing.
Task duration
Median coding time per task, excluding browser grading. Reported per agent because CLIs run differently.
Token usage (uncached)
Multi-step agents re-send cached conversation history on every turn, inflating raw totals. We report uncached tokens (total − cached) to measure new input and output per task.
Scheduled nightly runs
Automated suites (nightly-<date>-<agent>) run the full task set on a schedule. Headline numbers pool the latest nightly suite per agent.
Open-source benchmark
Guides, starter apps, prompts, and Playwright graders live in GoogleChrome/modern-web-guidance-src.

Browse all runs

Select a suite for per-task diffs and logs.