1. Relevance scoring model
Does the band an update is given predict whether it goes live? Baseline is the legacy gc3 scorer (score 3/2/1 read as High/Medium/Low); current is the Pinecone cosine model (High ≥ 0.647, Medium ≥ 0.583).
| Metric | Baseline legacy scorer |
Current cosine model |
Change | Legacy scorer, same month |
Target |
|---|---|---|---|---|---|
| % of High updates published | 16.1% | 12.7% | -3.4pt | 17.0% | not set |
| % of Medium updates published | 3.8% | 8.0% | +4.2pt | 4.8% | not set |
| % of Low updates published | 0.2% | 0.9% | +0.7pt | 0.2% | not set |
| % of Unscored updates published | 0.7% | 0.5% | -0.2pt | 1.1% | not set |
| False negatives: share of published that was Low/Unscored | 15.0% | 31.2% | +16.2pt | 11.9% | not set |
| Precision (High+Medium that went live) | 6.7% | 10.1% | +3.4pt | 8.0% | not set |
| Recall (published that was High+Medium) | 85.0% | 68.8% | -16.2pt | 88.1% | not set |
| Accuracy (High+Medium = publish) | 79.2% | 85.7% | +6.5pt | 77.0% | not set |
Decided items only: baseline 85,997 (1,500 live) ·
current 16,527 (369 live), plus
72,584 scored with no decision, excluded.
Accuracy is dominated by the Low band's volume — read precision and recall first.
Baseline by band — Legacy (pre-June 2026): Mar-May 2026
| Band | Decided | Went live | Accepted, not live | Rejected | % published | Share of all published |
|---|---|---|---|---|---|---|
| High | 4,465 | 719 | 434 | 3,312 | 16.1% | 47.9% |
| Medium | 14,449 | 556 | 376 | 13,517 | 3.8% | 37.1% |
| Low | 52,581 | 122 | 46 | 52,413 | 0.2% | 8.1% |
| Unscored | 14,502 | 103 | 41 | 14,358 | 0.7% | 6.9% |
Current by band — cosine model, 2026-09
| Band | Decided | Went live | Accepted, not live | Rejected | No decision | % published | Share of all published |
|---|---|---|---|---|---|---|---|
| High | 1,131 | 144 | 156 | 831 | 85 | 12.7% | 39.0% |
| Medium | 1,377 | 110 | 135 | 1,132 | 197 | 8.0% | 29.8% |
| Low | 12,136 | 106 | 102 | 11,928 | 62,152 | 0.9% | 28.7% |
| Unscored | 1,883 | 9 | 10 | 1,864 | 10,150 | 0.5% | 2.4% |
By vertical (baseline → current)
| Vertical | % High published | % Medium published | False-negative share | Precision | Recall | Published (n) |
|---|---|---|---|---|---|---|
| GC | 9.1% → 25.7% | 0.9% → 6.1% | 21.1% → 30.6% | 2.7% → 11.7% | 78.9% → 69.4% | 227 → 62 |
| PC | 17.3% → 17.3% | 6.7% → 8.7% | 20.5% → 30.6% | 9.2% → 12.2% | 79.5% → 69.4% | 747 → 209 |
| FS | 21.7% → 6.2% | 4.2% → 7.9% | 4.6% → 32.7% | 8.7% → 6.9% | 95.4% → 67.3% | 526 → 98 |
Monthly trend (by capture month)
| Month | Cosine model | Legacy scorer | ||||||
|---|---|---|---|---|---|---|---|---|
| % High pub. | Precision | Recall | Decided / live | % High pub. | Precision | Recall | Decided / live | |
| 2026-02 | — | — | — | 0 / 0 | 12.2% | 5.4% | 75.3% | 29,020 / 381 |
| 2026-03 | — | — | — | 0 / 0 | 14.6% | 7.5% | 72.7% | 34,447 / 557 |
| 2026-04 | — | — | — | 0 / 0 | 18.3% | 6.6% | 93.1% | 26,915 / 506 |
| 2026-05 | — | — | — | 0 / 0 | 15.7% | 6.3% | 91.3% | 24,635 / 437 |
| 2026-06 | 36.4% | 25.5% | 72.4% | 11,380 / 232 | 19.5% | 7.4% | 91.5% | 31,267 / 589 |
| 2026-07 | 24.8% | 19.7% | 56.5% | 25,975 / 584 | 16.5% | 6.6% | 93.5% | 25,975 / 584 |
| 2026-08 | 13.6% | 11.0% | 55.3% | 19,524 / 394 | 13.4% | 4.9% | 88.1% | 19,524 / 394 |
| 2026-09 | 12.7% | 10.1% | 68.8% | 16,527 / 369 | 17.0% | 8.0% | 88.1% | 16,527 / 369 |
| 2026-10 (partial) | 12.2% | 9.4% | 76.4% | 27,142 / 263 | 15.5% | 7.4% | 86.7% | 27,142 / 263 |
2. HS Gen draft generation
Do generated drafts go live, and how much do analysts change them? Revision is measured by diffing our draft body against the body that went live in Drupal.
| Metric | Baseline | Current | Change | Target |
|---|---|---|---|---|
| Publishability: drafts that went live | 61.1% | 48.5% | -12.6pt | not set |
| Near-verbatim (≥95% word similarity) of live drafts with a comparable published body |
2.9% | 11.0% | +8.1pt | not set |
| Minimal revision (<20% of words changed) | 21.3% | 42.4% | +21.1pt | not set |
| Median % of words changed | 44.0% | 24.0% | -20.0pt | not set |
| Factual accuracy proxy: fact recall share of the published numbers & names already in the draft |
76.0% | 81.7% | +5.7pt | not set |
| Drafts flagged by output guards figure grounding + scaffold leaks; most flagged drafts fail outright and are counted as failures instead |
0.0% | 0.8% | +0.8pt | not set |
| Editing time PROXY: draft ready → live, median queue time, not hands-on editing |
29.7 min | 72.8 min | +43.1 min | <= 5 minnot met |
| Editing time PROXY: live within 5 min | 26.6% | 2.6% | -24.0pt | not set |
| Generation failure rate | 6.2% | 4.6% | -1.6pt | not set |
Baseline: 2,170 drafts, 1326 live, 876 compared ·
Current: 645 drafts, 313 live, 290 compared.
The ≤5-minute editing target cannot be measured yet: nothing records when an analyst opens or edits a draft.
The proxy includes every hour a draft waits in the queue, so it overstates editing time.
By vertical (baseline → current)
| Vertical | Publishability | Near-verbatim | Minimal revision | Fact recall | Median ready → live | Drafts (n) |
|---|---|---|---|---|---|---|
| GC | 60.1% → 63.1% | 7.8% → 35.8% | 30.1% → 66.0% | 82.6% → 88.7% | 0.2 h → 0.3 h | 296 → 84 |
| PC | 61.8% → 45.4% | 2.4% → 6.1% | 19.0% → 37.0% | 75.5% → 80.7% | 0.4 h → 2.0 h | 1073 → 388 |
| FS | 60.5% → 48.6% | 1.9% → 4.2% | 21.8% → 37.5% | 74.6% → 78.7% | 0.7 h → 1.3 h | 801 → 173 |
Monthly trend (by draft month)
| Month | Drafts | Live | Publishability | Near-verbatim | Minimal revision | Median changed | Fact recall | Median ready → live | Failure rate |
|---|---|---|---|---|---|---|---|---|---|
| 2026-02 | 200 | 124 | 62.0% | — | — | — | — | 17.7 h | 16.7% |
| 2026-03 | 673 | 440 | 65.4% | 5.1% | 20.0% | 45.4% | 76.7% | 0.6 h | 11.8% |
| 2026-04 | 800 | 460 | 57.5% | 1.2% | 24.8% | 42.5% | 77.8% | 0.5 h | 3.1% |
| 2026-05 | 697 | 426 | 61.1% | 3.2% | 18.8% | 45.4% | 73.8% | 0.5 h | 3.7% |
| 2026-06 | 839 | 543 | 64.7% | 4.4% | 21.2% | 40.5% | 76.0% | 0.5 h | 5.1% |
| 2026-07 | 886 | 577 | 65.1% | 5.5% | 28.2% | 37.1% | 76.9% | 1.7 h | 3.4% |
| 2026-08 | 617 | 323 | 52.4% | 6.1% | 26.8% | 39.8% | 75.6% | 1.5 h | 4.8% |
| 2026-09 | 645 | 313 | 48.5% | 11.0% | 42.4% | 24.0% | 81.7% | 1.2 h | 4.6% |
| 2026-10 (partial) | 534 | 284 | 53.2% | 9.9% | 37.9% | 28.7% | 81.5% | 1.3 h | 3.4% |
What changed when
- 2026-02-24 HS Gen Prompt management + per-draft prompt snapshots; model claude-haiku-4-5 (unchanged since)
- 2026-05-15 HS Gen 8 analyst house-style rules + fabrication guard added to the prompt
- 2026-06-19 Relevance gc3 enrich pipeline launched (placeholder scorer, no cosines)
- 2026-06-24 HS Gen Radar Publish starts creating Drupal nodes (staging CMS)
- 2026-06-29 Relevance Cosine scorer live on Pinecone radar-drupal-nodes; bands 0.583 / 0.647 (never changed)
- 2026-07-21 Relevance Radar pill shows the legacy gc3 score instead of the cosine
- 2026-08-11 Relevance Radar pill back on the cosine bands
- 2026-08-19 Relevance In Scope two windows: High/Medium accumulate, Low/Unscored clear weekly
- 2026-08-21 HS Gen Radar Publish switched to the production CMS
- 2026-09-03 HS Gen Source publication date added to the prompt
How to read this
- Published means live in Drupal: a
Publisheddecision on the legacy Drupal form, or a Radar-created node whose published flag is set. Pressing Publish in Radar only creates an unpublished node, so it counts as "accepted, not live" until the node goes live. - Rates use decided updates only. An update nobody decided on may never have been seen (Low rows clear weekly), so it is not counted as a miss.
- Months are capture months (relevance) or draft months (HS Gen). Recent months keep changing as decisions land, and the partial month is greyed out.
- The comparison favours the legacy scorer on recall. Analysts working the legacy Drupal form see
content.scoreand triage by it, so what they publish is partly chosen by that score. The cosine model gets no such help, and the "legacy scorer, same month" column should be read with that in mind. - The model does not retrain. The cosine index grows nightly with every published Drupal article, so improvement shows up as a trend, not at dated releases. Rejected updates never enter it.
- Known gaps: scores are overwritten on re-score (the band shown is the latest one); the
drupal_nodesmirror misses ~12% of published nodes, so revision measures use a sample; regenerating a draft overwrites the first version; no model-version column exists, so the change list is from git history. - Targets live in
src/dashboard/hs_gen/board_targets.json. "not set" means nobody has agreed one yet.