fab-kit findings · 2026-08-10 · experiment report, part 2
Two draft intakes — rvza (config-show-compose-defaults, a surgical Go fix) and ttff (agy-interactive-pane-capability, a prose-heavy repo-wide sweep) — were each run to a draft PR three times through the full fab-fff pipeline: on kimi (kimi --auto, k3), on codex (gpt-5.6-sol @ xhigh), and on claude (claude-opus-5 @ high). The kimi and codex arms ran as one concurrent batch (~11:25 IST); the claude arm ran ~2.5 hours later as its own two-run batch — with less machine contention, which makes its wall-clock numbers conservative. This page extends the earlier two-provider report; the rubric, scoring protocol, and kimi/codex data are unchanged.
Kimi is ~2.6× faster than either; claude and codex are effectively tied on quality (9.15 vs 9.05 mean), each winning one change; codex ships the leanest diffs, claude the largest.
Claude took the small surgical change outright (9.20 — the highest rvza score, catching a vacuous-test regression the others shipped). Codex kept the sweep-heavy crown (9.45 vs claude's 9.10, which lost points only to prose polish). Kimi stayed the speed pick: third on quality both times, but at 2.5–2.6× the pace. Diff size, the new metric, splits cleanly: codex is the most economical on both changes; claude writes 47–55% more lines than codex — extra tests, guards, and docs that scorers judged substantive rather than churn.
Same intake, same pipeline per triple — only the worker provider differs. All six PRs are draft, CI-green, produced without human intervention, and all six have now completed review-pr.
| PR | Arm | Change | Worker roster | Wall-clock | Rework | Copilot | Diff (lines · files) | Score |
|---|---|---|---|---|---|---|---|---|
| #570 | kimi | rvza | kimi --auto (k3), all stages, pane | 13m 35s | 0 | 0 | 362 · 9 | 8.40 |
| #572 | codex | rvza | sol@xhigh; hydrate sol@high; ship luna@low | 33m 27s | 0 | 1 | 283 · 9 | 8.65 |
| #575 | claude | rvza | opus-5@high pane; ship sonnet@med + review-pr opus@high native | 34m 45s | 0 | 0 | 439 · 12 | 9.20 |
| #571 | kimi | ttff | kimi --auto (k3), all stages, pane | 28m 04s | 1 | 5 | 706 · 25 | 8.65 |
| #573 | codex | ttff | sol@xhigh; hydrate sol@high; ship luna@low | 71m 42s | 2 | 0 | 686 · 26 | 9.45 |
| #576 | claude | ttff | opus-5@high pane; ship sonnet@med + review-pr opus@high native | 81m 57s | 2 | 4 | 1036 · 25 | 9.10 |
Stage spans reconstructed from .fab-dispatch/ record and result-file timestamps; x-axis is minutes since each run's worktree creation. Gaps between segments are orchestrator triage time; only the final rework cycle's files survive, so a reworked run's apply segment includes its interleaved failed reviews. The vertical PR tick is the measurement endpoint; dashed-gray segments after it are the post-PR Copilot review-pr stage. Dash-outlined segments are estimated from commit timestamps (stages that ran on the native seam and left no dispatch record).
| Run | Worktree born | Apply span | Review | Hydrate | PR | Review-pr |
|---|---|---|---|---|---|---|
| #570 kimi·rvza | 11:24:57 | 11:27:50–11:31:43 | 11:32:44–11:34:14 | 11:35:05–11:36:06 | 11:38:32 | 11:39:44–11:44:59 |
| #572 codex·rvza | 11:25:00 | 11:28:46–11:43:57 | 11:45:06–11:50:37 | 11:52:16–11:55:44 | 11:58:27 | 12:04:23–12:13:30 |
| #575 claude·rvza | 13:46:18 | 13:48:49–14:00:12 | 14:01:02–14:08:24 | 14:10:57–14:18:47 | 14:21:03 | ~14:21–14:29:55 (native) |
| #571 kimi·ttff | 11:25:03 | 11:28:15–11:45:46 (×1) | 11:46:16–11:48:01 | 11:48:59–11:50:29 | 11:53:07 | 11:54:13–12:03:36 |
| #573 codex·ttff | 11:25:05 | 11:28:02–12:19:29 (×2) | 12:20:10–12:27:38 | 12:28:22–12:32:53 | 12:36:47 | ~12:37–12:41:25 (est) |
| #576 claude·ttff | 13:46:21 | 13:48:26–14:44:08 (×2) | 14:44:28–14:58:27 | 15:00:29–15:05:07 | 15:08:18 | 15:08:18–15:17:27 (native) |
Lines touched (insertions + deletions) and files changed per PR, from git diff main...branch --stat. Size alone is not a quality signal — the scorers judged where the extra lines went (see verdicts) — but it is a real review-cost and merge-risk signal for the human who reads the PR.
Codex is the most economical on both changes — its rvza diff is 22% smaller than kimi's and 36% smaller than claude's for a near-identical score, and its scorer singled out "deletes a special case rather than adding a branch" as the diff's defining trait. Claude is consistently the largest (+55% vs codex on rvza, +51% on ttff): the extra lines are three additional tests, a sweep-regression guard, beyond-intake doc sweeps, and a hazard write-up — substance the scorers credited (both claude runs scored 9–10 on test quality), but also the most review surface for a human to read. Kimi sits between, skewing additive (+326/−36 on rvza) where codex balances additions against deletions.
Each PR scored blind-to-its-siblings by an independent sub-agent against the same anchored 0–10 rubric, grading the final branch state against the intake, with mandatory file:line evidence and scorer-run tests. Weights: intake fidelity 20% · correctness 20% · diff hygiene 20% · sweep completeness 15% · test quality 15% · prose & docs 10%. Bars per dimension: kimi, codex, claude.
Top strengths and weaknesses per PR, as reported by each scoring sub-agent. Kimi/codex cards repeat the earlier report for side-by-side reading.
"The implementation is the right one … the one real gap is test coverage of the empty-repo case … minor hygiene noise keeps this short of the top band."
"A tight, well-targeted change that does exactly what the intake specified and nothing more … what keeps it out of the top band is self-review rather than judgment."
"Nine lines of real substance … correct in the way that matters most … the sweep is the strongest part. The only things I would change are cosmetic."
"Falls short on exactly the axis the intake flagged as the repo's #1 rework cause … widening the sweep grep to phrase classes would take this to a 9-plus."
"A near-exemplary execution of a change that is 80% sweep discipline and 20% code … what keeps it off a perfect score is cosmetic."
"A strong, complete execution … the sweep is the standout, plus a new regression guard that keeps it that way. What holds it short of top marks is prose polish … nothing here would block a merge."
Claude took the surgical Go change (9.20, the only rvza run to catch the vacuous-test regression); codex kept the sweep-heavy change (9.45 vs claude's 9.10). Their means are statistically indistinguishable (9.15 vs 9.05). Kimi placed third on both, held back by test coverage on rvza and the sweep miss on ttff.
Claude is marginally slower than codex on both changes (34m45s vs 33m27s; 81m57s vs 71m42s) — and the claude batch ran with only two concurrent pipelines instead of four, so its numbers are if anything flattered. The ordering kimi ≪ codex ≈ claude is robust.
The pattern repeats across both changes: codex writes the leanest diff (its rvza change is net-negative on implementation lines), claude the largest (+51–55% vs codex), kimi in between and skewing additive. Claude's extra lines bought real things — a sweep-regression guard, three extra tests, a documented footgun — but the review surface a human must read grows with it. If diffs-a-human-reviews is the constraint, codex; if defense-in-depth is, claude.
Kimi shipped two stale claims phrased outside the plan's grep tokens ("headless grammar only"). Claude's run surfaced a second invisible class: cycle-1 rework was a contrastive phrasing miss — "unlike agy" carries the retired claim by contrast and no literal grep finds it. Claude's internal reviewer also pre-fixed the exact two stale lines kimi's PR still carries, and its PR ships a regression test pinning the retired literals out of the rendered reference. Sweep-verification greps should target phrase classes AND contrast markers (unlike X, except X, X alone).
Kimi's reviewer accepted ttff after 1 cycle; Copilot then raised 5 comments. Codex's reviewer forced 2 cycles to a 0-finding pass; Copilot found nothing on either codex PR. Claude's reviewer also forced 2 cycles and was the only one to produce substantive findings on the small change too (the vacuous-test catch) — Copilot's 4 comments on #576 were one cosmetic wording class. Internal review depth, not apply-stage brilliance, is where the quality gap is manufactured.
Three different provider CLIs — kimi and codex via pane workers, claude via pane + native seams — each drove intake-to-review-pr unattended to CI-green draft PRs, including rework choreography and Copilot-comment triage. The provider choice is a pure cost/quality/speed/diff-size knob.
One sub-agent per PR with an identical prompt: read the project constitution and quality/review policies, the intake and plan, the full main...HEAD diff; run go test + gofmt in the PR's own worktree; score six anchored dimensions 0–10 with mandatory file:line evidence; grade against the intake, never against sibling PRs. Anchors: 10 flawless · 8–9 minor nits · 6–7 one real gap · 4–5 significant gap · ≤3 major failure. One prompt asymmetry: the #576 scorer's instructions included the phrase-class grep lesson learned in round 1; the round-1 ttff scorers were not given it — the #571 scorer discovered the phrase-class misses on its own initiative and the #573 scorer independently verified variant phrasings, so the sweep dimension was in practice examined at comparable depth across all three arms.
| Dimension | Weight |
|---|---|
| Intake fidelity — every "What Changes" item, nothing beyond scope | 20% |
| Correctness — logic read directly; tests run by the scorer | 20% |
| Diff hygiene — minimal diff, pattern conformance, commit/PR quality | 20% |
| Sweep completeness — mirror classes, aggregate specs, memory | 15% |
| Test quality — pin-by-value, coverage relocated not deleted | 15% |
| Prose & docs — present-truth framing, no transition narration | 10% |