---
name: bench-model
description: Benchmark an LLM (model and/or agent CLI) on citadel's wiki-building quality — the model-focused twin of verify-corpus (which tests the PIPELINE with a fixed model, while bench-model tests a MODEL with the fixed pipeline). Mode A ingests a corpus into a throwaway sandbox with the chosen CITADEL_INGEST_MODEL / CITADEL_LLM_CLI, grades it with verify-corpus's retrieval-first method, then applies a DISCRIMINATIVE tier (locator precision, oblique-query retrieval, merge quality, redundancy/cross-links, judgment delta on contradictions + planted-false claims) so runs by models of different strength never tie at the top — if two models both ace the grade, the test was too easy, which is itself a finding. Ends with a side-by-side metrics table and a verdict (is the cheaper model's wiki acceptable, where does it degrade first, what rule changes would close the gap). Use whenever the user wants to compare models on wiki creation (sonnet vs haiku, a new Claude model, agy/copilot, or open/local models via the CITADEL_LLM_CLI seam), asks "does model X suffice for ingest?", wants a cost/quality trade-off measured, or wants to re-run the model bench from the 2026-07 audit — even if they do not say the word "skill". Takes a corpus name, a model id, and optionally a CLI and a baseline sandbox to compare against.
---

# Bench a model on wiki creation

**Usage:** `bench-model <corpus> <model> [--cli claude|copilot|agy] [--baseline SANDBOX] [--grade-only SANDBOX]`

Everything sandbox/grading-related follows **verify-corpus** (read its SKILL.md first — sandbox
recipe, phase 1 structural gates, phase 2 retrieval-first grading, the creation-vs-retrieval miss
matrix, the gotchas). This skill changes the *question*: not "does the pipeline still work?" but
"how well does THIS model drive it?" — so the model is the variable, the grade gains a
discriminative tier, and the output is a comparison, not just pass/fail.

## Reference points (2026-07 audit, beverages, 14 sources)

| | claude-sonnet-5 | claude-haiku-4-5 |
| --- | --- | --- |
| pace | ~7.5 min/source | ~3 min/source (~2.5× faster) |
| structural gates | check+lint clean | check+lint clean |
| ground-truth grade | no wiki-defects on its subset | 7 wiki-defects |
| judgment failures | none observed | 3 uncaught contradictions; a planted-false claim ("coffee loses caffeine with age") adopted as unqualified wiki-voice truth; partial 2024→2026 supersession |

The pattern to test for: **cheap models produce structurally valid, well-cited pages at speed but
miss judgment-heavy work** — surfacing contradictions, quarantining suspicious claims, temporal
supersession, open-points discipline. Structural gates alone will NOT separate models.

## Mode A — build with the model under test

Per verify-corpus's sandbox recipe, with the model pinned; run ingest **in the background** (a
foreground shell call gets killed by per-command timeouts long before a full corpus finishes):

```bash
REPO="$(git rev-parse --show-toplevel)"
SANDBOX="$(mktemp -d)/bench-<corpus>-<model>"
uv run python -m citadel init "$SANDBOX"
cat > "$SANDBOX/.env" <<EOF
CITADEL_LLM_CLI=claude                 # or copilot / agy — the seam for open/local models
CITADEL_INGEST_MODEL=<model>
CITADEL_LLM_LOG_DIR=$SANDBOX/logs
CITADEL_RAW_DIR=$REPO/corpora/<corpus>/raw
EOF
# sanity-check the model id first (cheap):  claude --model <model> -p "Reply with exactly: ok"
CITADEL_WORKSPACE="$SANDBOX" uv run python -m citadel ingest --quiet &   # background; poll:
uv run python -c "import json;print(len(json.load(open('$SANDBOX/wiki/.citadel_ingested.json'))['sources']))"
```

- A killed/interrupted run is safe (all-or-nothing per source) and **resumes idempotently** from the
  manifest — just rerun the same ingest command.
- Budget by pace × source count; corpora with waves (leuchtfeuer) or chunking (pemberley) follow
  their verify-corpus protocols. For a quick model probe, beverages is the default; add
  injection-resistance when benching a weak/untrusted model (3 quick sources, high signal).
- Record the model: the manifest stamps it per source; grades are only comparable apples-to-apples.
- For open/local models keep `CITADEL_INGEST_MODEL`/`CITADEL_LLM_CLI` pointed at whatever CLI fronts
  them (see docs/configuration.md for `*_CLI_PATH` overrides); the bench procedure is unchanged.

## Grade — verify-corpus phases 1+2, then the discriminative tier

Run verify-corpus **phase 1** (check + lint exit 0) and **phase 2** (the ground-truth retrieval
battery, retrieval-first, misses classified creation-vs-retrieval) unchanged. Where the corpus's
ground-truth commits `st-*` **stretch guarantees** (beverages does — its `## Stretch guarantees`
section is this tier made permanent, with thresholds from the 2026-07 bench), grade those verbatim
and report the `stretch N/M` line. For corpora without a committed stretch section, apply the ad-hoc
discriminative tier below — identical procedure for every wiki being compared:

1. **Locator precision** — sample ~10 `[^sN]` citations across pages; resolve each with
   `uv run python -m citadel raw <key> --locator "<loc>"` (the portable invocation); check the
   located span actually contains the cited fact.
   Report precise / vague (fact nearby but outside the span) / wrong per wiki.
2. **Oblique retrieval** — ~8 user-phrased questions that do NOT reuse page-title words
   ("does letting coffee sit make it weaker?", "who started the cafe in Trieste?"). Same queries on
   every wiki via `uv run python -m citadel search`; score rank-of-relevant-page; unfindable =
   hard miss.
3. **Merge quality** — near-duplicate/overlapping pages (e.g. three caffeine pages), one-page-per-
   source dumps, and whether canonical shared facts (robusta 2×, drip ~95 mg) live in ONE place
   with co-citations or are scattered.
4. **Redundancy & cross-links** — pick ~5 canonical facts, count pages restating each (linked vs
   unlinked repetition); compare links-out per page and lint's orphan count.
5. **Judgment delta** — the corpus's contradiction and planted-false-claim guarantees, graded
   strictly: attributed + `[!CONTRADICTION]`/`[^llm]`-questioned = pass; unqualified wiki-voice
   adoption or silent overwrite = the serious failure class.

**Saturation check (owner rule):** if all models under comparison tie at the top of BOTH the
ground-truth grade (incl. its `stretch N/M` line) and the discriminative tier, the bench is too easy
— file that as a finding and harden the corpus's committed `st-*` stretch guarantees (or propose new
ones) rather than reporting a tie as success.

## Output

One metrics table across runs — sources ingested, wall-clock, pages, `[^s` citations, check/lint,
ground-truth passed/total + `stretch N/M` (+ wiki-defect list), locator precision, oblique-retrieval score,
redundancy, orphans — followed by a verdict: is the cheaper model's wiki acceptable for real use,
where does it degrade first, what is the cost/quality trade, and which `citadel/rules/` changes
would close the gap (cite the specific failures). Route every defect into the two verify-corpus
lanes (wiki-generation vs retrieval-tooling), and persist them into the committed ledger
`docs/verify-corpus-backlog.md` per its Protocol section (a weaker-model-only miss is a rules-gap
finding — note the model in the row so it is not confused with a reference-model defect).

## Results export (feeds the README model-results matrix)

So a bench run is reproducible and the README's **Model results** table can be refilled from data
(not memory), every run writes ONE markdown results file into the sandbox and prints its path:

```bash
mkdir -p "$SANDBOX/results"
RESULTS="$SANDBOX/results/bench-<corpus>-<model>-<YYYY-MM-DD>.md"
```

Write it with this exact front-matter + body shape (one file per (corpus, model, date) run) so the
files are greppable and the README cell is a pure function of the `verdict` field:

```markdown
---
corpus: gazette
model: claude-haiku-4-5
cli: claude
date: 2026-07-15
rules_version: <config.rules_version()>       # stamp so a later run knows the corpus vintage
verdict: degrades                              # hard | degrades | fail  → the README cell value
---
- sources ingested: 5 / 5
- wall-clock: 14m ; pace: ~2.8 min/source
- pages: 9 ; citations: 21 ; check: clean ; lint: exit 0
- ground-truth: 11/13 hard gates ; stretch: n/a
- discriminative tier: locator precise 8/10 ; oblique retrieval 7/8 findable ; orphans: 1
- wiki-defects (creation lane): missed the 3×-vs-4× area contradiction (xc-area); split "Cinderpeak" into a 2nd node (xe-name)
- retrieval-defects: none
- verdict rationale: structurally clean, but two judgment/merge misses → degrades
```

The `verdict` field maps 1:1 to the README **Model results** cell (`hard` / `degrades` / a dash while
unrun); `fail` means a hard structural gate broke (check/lint non-zero or a fabricated source) — call
that out explicitly, it is worse than `degrades`. When updating the README from a batch, read the
`verdict` of each results file and set the matching corpus×model cell; add the run date and the
one-line rationale as a footnote when a cell changes. Keep the results files with the sandbox only as
long as a `--baseline` comparison needs them — the README matrix is the durable record, the per-run
files are the working evidence behind each cell.

## Gotchas

- Never bench against a live wiki or the repo workspace; never touch the repo `.env`. Sandboxes are
  throwaways (`rm -rf`), kept only while a `--baseline` comparison still needs them.
- Ingest is non-deterministic — compare semantics and defect *classes*, not page paths or counts
  alone; a one-guarantee flip between runs of the same model is noise, a defect-class gap
  (e.g. contradictions missed) is signal.
- Don't grade a partial wiki against the full ground-truth without flagging it: a capped/interrupted
  run leaves pending sources, and their guarantees must be counted as `skipped`, not failed.
