MWG CLI: Search quality over time

Agents discover Modern Web Guidance through modern-web-guidance search before retrieving a guide. Quantifying retrieval quality, token efficiency, and cold-start latency across our chunking, BM25 lexical matching, and embedding pipeline keeps the gateway to our guides fast and reliable (#3106).

Milestone benchmark Unlike the main eval dashboard, which updates nearly every day with new agent runs, this page updates when changes land in the search pipeline—currently few and far between.

1Search quality across milestones

Each milestone replayed on the 192-guide corpus · held-out test splits (#3106) · 95% CI

Queries All queries 2–5 words 6+ words Agent queries

X axis By milestone By date

How we score search

Hit@K Did a right guide show up?: Share of searches where at least one right leaf guide appears anywhere in the K results. Rank 1 and rank 10 get the same credit; a missing second guide costs nothing.
Recall Did all the right guides show up?: Share of a query’s right leaf guides that appear in the K results. A query that needs two guides and gets one scores 100% on Hit@K but 50% on recall.
MRR How close to #1 was the first right guide?: Mean reciprocal rank: 1 ÷ the rank of the first right guide. #1 = 100%, #2 = 50%, #5 = 20%, #10 = 10%. It appears in PR benchmarks; it isn’t charted here.
Leaf nDCG@K Did all the right guides rank near the top?: Recall weighted by rank: a right guide at rank r earns 1 ÷ log₂(r + 1), divided by the best possible score. Index pages like css earn nothing.
Signal-to-noise Quality minus token waste: 70% recall + 30% leaf nDCG, minus 1.4 points per 100 tokens spent on wrong guides (4.5 beyond the first 500). An empty result counts as 1,500 wasted tokens: the list fallback.

On Wild-v2 test (865 queries), top-10 was the biggest step (+7.4 pp to 93.5%), then BM25 stemming (#3102, 94.5%), chunk metadata (#3172, 94.7%), and #3089 static 256-d query vectors × build-time gemini-embedding-001 chunks (95.4% Hit@10, 3.2× faster) climbed monotonically. On the held-out Agent test split (494 queries), #3089 reaches 99.4% Hit@10.

Leaf nDCG also rewards ranking the right guide first and finding its siblings. Across the post-#3048 stack, Leaf nDCG climbs from 70.6% (#3048) → 71.0% (#3102) → 72.3% (#3172) → 75.0% (#3089, +2.7 pp over MiniLM + #3172); online gemini-embedding-001 reaches 81.3% and online gemini-3.8-flash catalog search reaches 91.6%.

Signal-to-noise charges for tokens spent on wrong guides and for list fallbacks. On Wild-v2 test, v0.0.195 drops to 57.0% because empty results force fallbacks; compact top-10 lifts it to 78.1% (#3048) → 78.8% (#3102) → 79.4% (#3172) → 80.6% (#3089); online gemini-embedding-001 and gemini-3.8-flash reach 84.8% and 90.4%.

2Milestones compared

Every system on this page · scores on Wild-v2 test (865 leaf-answerable of 2,000 held-out queries, #3106)

Milestone	When	What changed	Output	Hit@K	Leaf nDCG	Signal-to-noise

3Why K=10 is the sweet spot

Wild-v2 test (865) at K=1..20 (0% unjudged) · signal-to-noise subtracts a tax for tokens spent on wrong guides

Metric Signal-to-noise Hit@K Leaf nDCG

Verbose output peaks at K=5 (71.5% SNR) because five results already cost 524 tokens. Compact output spends the same ~529 tokens on ten results, lifting SNR to 78.1% (93.5% hit) on #3048 and 79.0% (93.6% hit) on #3089. Past K=10, extra results add a little recall and a lot of noise.

4Price vs. quality: runtime, memory, and package size

Cold search, re-measured back to back · bubble area = Peak RSS · up and to the left is better

Quality axis Hit@K Signal-to-noise

Cold search3.2× faster

770→240ms

Peak RSS bubble area−58%

346→145MB

npm tarball−64%

23.1→8.3MB

Stdout tokens: 507 → 528 tok for 2× the results (top-5 verbose → top-10 compact).

Every TF.js release ran 0.7–0.8 s cold. Search fixes (#2052 + #3048) added +12.9 pp of Hit@K at that cost, then #3089’s static embedder (stacked on #3102 + #3172) cut cold search 3.2× while lifting Hit@10 to 95.4% and Leaf nDCG to 75.0%.

5Methodology and corpus details

Who decides “right” on Wild-v2 and Agent (#3106)

Both evaluation datasets ship in PR #3106. Candidate pools are graded 0–2 by gemini-3.8-flash (rubric v1, temperature 0) from guide cards (id, title, description, features, headings), calibrated against 69 blind human ratings achieving 0.78 quadratic-weighted Cohen’s κ, 93.3% binary hit agreement, and 92.8% answerability agreement. A query counts in leaf metrics when at least one non-discipline leaf guide receives grade 2:

Anti-leakage splits: Strictly partitioned to prevent data leakage: zero stemmed token-bag overlap between wild-v2 dev and test splits; 70 connected-component guide clusters on agent so zero guides are shared between dev and test.
Wild-v2 test: 2,000 held-out queries (of 4,000 in wild-v2, qrels SHA 8eff50545bdd); 865 are leaf-answerable (702 short 2–5 words, 159 long 6+ words, 4 one-word). Top-10 across all milestones plus ranks 11–20 across the K-curve systems are 100% judged (0% unjudged; +8,131 top-10 and +19,164 rank 11–20 overlay judgments).
Agent test: 514 held-out task queries (of 1,029 leakage-filtered agent searches in agent-v1, qrels SHA d5fb45b5a0b4), partitioned by connected guide components into 70 test clusters so zero guides are shared with dev; 494 are leaf-answerable (0% unjudged @10 across all milestones; +2,418 overlay judgments). CIs resample whole expected-guide clusters.
Leaf guides & tokens: Only grade-2, non-discipline guides count as right. Tokens: 90/result before #824, 105 verbose after #824, 53 compact from #3048. An empty result costs 1,500 fallback tokens (the list catalog).

Method

Section 1: Every milestone is replayed in cumulative landing order on the 192-guide corpus (including #2130’s ui-components/dialog; 192/1943/1943/1773 chunks across m1–m4; 1809 enriched chunks from #3172/#3089); top-5 until #3048, top-10 from #3048. #2052 and #3048 apply the 0.30 MiniLM cosine floor (1 empty on scored Wild-v2 test). #3102 consolidates Lucene EnglishMinimalStemmer in tokenizeBm25(), the BM25-aware 0.30 similarity gate, and sub-floor dense exclusion on BM25-only rescues. #3172 (stacked on #3102) prepends guide title, category, and raw + unslugged web-feature IDs to every chunk for MiniLM and BM25 indexing.
#3089 static: Asymmetric static model using 4-step Tokenlearn distillation: 29,165-word whole-token web vocabulary distilled from gemini-embedding-001 (RETRIEVAL_QUERY, Matryoshka 256-d), fine-tuned with 12 epochs of Adagrad over 32,915 holdout-scrubbed queries on cosine loss, and quantized to folded per-row Int8. Scored asymmetrically against build-time 256-d gemini-embedding-001 (RETRIEVAL_DOCUMENT) chunk vectors from the committed cache. Why asymmetric? Summing 3 query words stays sharp in 256-d space (mean pairwise cosine 0.28), whereas averaging ~300 words across a markdown chunk pulls document vectors into a narrow 0.970 cosine cone on the unit sphere (where “best pizza in new york” scores 0.899 against <dialog>). Build-time transformer chunk vectors eliminate cone collapse while keeping runtime query embedding at ~30 µs. Search only returns [] when dense top-1 < 0.30 and BM25 has no matches (0 empty on scored test).
Baselines: Reference columns compare online 768-d gemini-embedding-001 and online gemini-3.8-flash LLM catalog search (1 query per call over the 192 {id, description} entries, ~8.1k prompt tokens, temperature: 0, thinkingBudget: 0; note that gemini-3.8-flash also graded the qrels from fuller guide cards with headings, while Agent test queries also carry independent task expectedGuide labels: 93.0% known-answer Hit@10 vs. 96.1% on gemini-embedding-001 and 95.5% on #3089).
Section 3: Scores K=1..20 on the 865 Wild-v2 test queries (0% unjudged through K=20).
Section 4: Cold search, medians of 6 queries measured back to back.