fab-kit findings · 2026-08-10 · experiment report, part 3 (final)
The same two draft intakes — rvza (config-show-compose-defaults, a surgical Go fix) and ttff (agy-interactive-pane-capability, a prose-heavy repo-wide sweep) — were each run to a draft PR through the full fab-fff pipeline on four worker providers: kimi (kimi --auto, k3), codex (gpt-5.6-sol @ xhigh), claude (claude-opus-5 @ high), and agy (gemini-3.1-pro-high, all stages on pane workers). This page supersedes the two-provider and three-provider rounds; rubric and prior data are unchanged. A pleasing recursion: the agy arm ran on the very interactive_command pane grammar that the ttff change — which it was implementing — shipped to main hours earlier.
Kimi is the speed pick; claude and codex tie for quality, each winning one change; agy trails clearly on quality (6.63 mean) despite mid-pack speed and the leanest diffs.
Agy's Go work is competent — both scorers called the core implementation "close to ideal" / "clean and pattern-faithful" — but it loses where this repo's bar is highest: finishing discipline. It missed the constitution-required SPEC mirror on rvza, left four stale claims on ttff, wrote an unperformed probe into memory as fact, and hand-edited frozen generated history. Its lean diffs are partly lean because the sweep work is missing — unlike codex, whose lean diffs are complete.
Same intake, same pipeline per quadruple — only the worker provider differs. All eight PRs were produced without human code intervention and completed review-pr. The kimi/codex pair ran as one concurrent batch (~11:25 IST), claude ~13:46, agy ~17:52; later batches had less machine contention. The codex PRs (#572, #573) were subsequently picked and merged; the agy branches are based just before those merges, so their diffs are scored against their own base.
| PR | Arm | Change | Worker roster | Wall-clock | Rework | Copilot | Diff (lines · files) | Score |
|---|---|---|---|---|---|---|---|---|
| #570 | kimi | rvza | kimi --auto (k3), all stages, pane | 13m 35s | 0 | 0 | 362 · 9 | 8.40 |
| #572 | codex | rvza | sol@xhigh; hydrate sol@high; ship luna@low | 33m 27s | 0 | 1 | 283 · 9 | 8.65 |
| #575 | claude | rvza | opus-5@high pane; ship sonnet@med + review-pr opus@high native | 34m 45s | 0 | 0 | 439 · 12 | 9.20 |
| #578 | agy | rvza | gemini-3.1-pro-high, all stages, pane | 35m 55s† | 0† | 2 | 190 · 9 | 6.80 |
| #571 | kimi | ttff | kimi --auto (k3), all stages, pane | 28m 04s | 1 | 5 | 706 · 25 | 8.65 |
| #573 | codex | ttff | sol@xhigh; hydrate sol@high; ship luna@low | 71m 42s | 2 | 0 | 686 · 26 | 9.45 |
| #576 | claude | ttff | opus-5@high pane; ship sonnet@med + review-pr opus@high native | 81m 57s | 2 | 4 | 1036 · 25 | 9.10 |
| #579 | agy | ttff | gemini-3.1-pro-high, all stages, pane | 53m 09s | 1 | 5 | 507 · 23 | 6.45 |
† #578 had zero review-fail rework cycles but spent the pipeline's single recovery restart: its first apply worker completed the work without writing the required result file (a worker-contract miss no other arm produced), and the ~16-minute lost attempt is inside its wall-clock.
Stage spans from .fab-dispatch/ record and result-file timestamps; x-axis is minutes since each run's worktree creation. Gaps between segments are orchestrator triage; a reworked run's apply segment includes its interleaved failed reviews (only the final cycle's files survive). The PR tick is the measurement endpoint; dashed-gray segments after it are the post-PR Copilot review-pr stage; dash-outlined segments are estimated from commit timestamps (native-seam stages with no dispatch record).
| Run | Worktree born | Apply span | Review | Hydrate | PR | Review-pr |
|---|---|---|---|---|---|---|
| #570 kimi·rvza | 11:24:57 | 11:27:50–11:31:43 | 11:32:44–11:34:14 | 11:35:05–11:36:06 | 11:38:32 | 11:39:44–11:44:59 |
| #572 codex·rvza | 11:25:00 | 11:28:46–11:43:57 | 11:45:06–11:50:37 | 11:52:16–11:55:44 | 11:58:27 | 12:04:23–12:13:30 |
| #575 claude·rvza | 13:46:18 | 13:48:49–14:00:12 | 14:01:02–14:08:24 | 14:10:57–14:18:47 | 14:21:03 | ~14:21–14:29:55 (native) |
| #578 agy·rvza | 17:51:58 | 17:57:59–18:19:33 (restart 18:13:55) | 18:21:02–18:22:57 | 18:23:59–18:25:38 | 18:27:53 | 18:29:11–18:39:27 |
| #571 kimi·ttff | 11:25:03 | 11:28:15–11:45:46 (×1) | 11:46:16–11:48:01 | 11:48:59–11:50:29 | 11:53:07 | 11:54:13–12:03:36 |
| #573 codex·ttff | 11:25:05 | 11:28:02–12:19:29 (×2) | 12:20:10–12:27:38 | 12:28:22–12:32:53 | 12:36:47 | ~12:37–12:41:25 (est) |
| #576 claude·ttff | 13:46:21 | 13:48:26–14:44:08 (×2) | 14:44:28–14:58:27 | 15:00:29–15:05:07 | 15:08:18 | 15:08:18–15:17:27 (native) |
| #579 agy·ttff | 17:51:59 | 18:01:37–18:34:45 (×1) | 18:35:42–18:38:16 | 18:39:09–18:42:50 | 18:45:08 | 18:46:12–19:01:46 |
Lines touched (insertions + deletions) and files changed per PR, each against its own merge base. Size alone is not a quality signal — the scorers judged where the lines went — but it is a review-cost signal, and this round adds a twist: a lean diff can be lean because the work is missing.
Agy is the smallest on both changes — but its missing SPEC-mirror update (rvza) and four unswept claims (ttff) account for part of the shortfall: some of those absent lines are the work. Codex remains the benchmark for complete economy: second-leanest on both changes with a verified-complete sweep. Claude stays the largest (defense-in-depth: extra tests, guards, hazard docs the scorers credited); kimi sits in between, skewing additive.
Each PR scored blind-to-its-siblings by an independent sub-agent against the same anchored 0–10 rubric, grading the final branch state against the intake, with mandatory file:line evidence and scorer-run tests. Weights: intake fidelity 20% · correctness 20% · diff hygiene 20% · sweep completeness 15% · test quality 15% · prose & docs 10%. Bars per dimension: kimi, codex, claude, agy.
The six kimi/codex/claude cards are unchanged from the three-provider report; here are the two new ones in full, plus the two change-winners for reference.
"The implementation itself is close to ideal … what holds it back is finishing discipline rather than engineering judgment. A reviewer would merge the Go change happily and send the docs and dead branch back for one round."
"The mechanical change is correct and well-executed … Merging this is safe for the binary but ships documentation that contradicts it in four places, so a follow-up sweep is required rather than optional."
"Nine lines of real substance … the sweep is the strongest part. The only things I would change are cosmetic."
"A near-exemplary execution of a change that is 80% sweep discipline and 20% code … what keeps it off a perfect score is cosmetic."
Four data points now: the strictness of each arm's internal reviewer — which is the same model as its worker — predicts final PR quality better than apply-stage output does. Codex and claude reviewers forced 2 rework cycles and produced 9+ PRs; kimi's reviewer accepted after 1 cycle and shipped a sweep miss; agy's reviewer passed a PR whose SPEC mirror contradicted the binary (rvza) and found 0 findings on a ttff cycle that still carried four stale claims. The binding constraint in this pipeline is the critic, not the author.
Beyond ordinary sweep misses, agy produced three failure classes no other arm did: an apply worker that finished its work but skipped the result-file contract (costing the recovery restart), a memory Design Decision claiming a probe that was explicitly deferred ("Probed … Verbatim"), and hand-edits to frozen generated history. The first is an operational nuisance; the latter two poison the knowledge base — they are the exact failure the FKF present-truth rules exist to prevent, and they argue for keeping agy on a shorter leash than a score of 6.6 alone would suggest.
The diff-size metric needed one round to show its trap: agy's diffs are the smallest on both changes, but part of the missing volume is the missing work. Codex holds the useful crown — second-leanest with verified-complete sweeps. Read diff size jointly with sweep completeness, never alone.
Token misses (none this round — every arm greps canonical vocabulary correctly), phrase-class misses ("headless grammar only" — kimi and agy), and contrastive misses ("unlike agy" — claude's rework, agy's #579). Verification greps must cover all three; the strongest observed defense is codex/claude-style re-derivation of the sweep class plus claude's regression-guard test pinning retired literals out of rendered output.
Agy is genuinely the second-fastest engine — its restarted rvza apply ran just 5m 38s, and per-stage spans sit between kimi's and codex's. But its contract miss cost 16 minutes of its rvza wall-clock, and speed is worth little at a 6.6 quality level in a repo whose review gates treat sweep misses as must-fix: the rework it avoids up front comes back as human-triggered follow-up later.
Small surgical change: kimi for economics, claude for the best absolute result. Wide prose sweep: codex (lean, complete, merged) or claude (thorough, defense-in-depth). Agy, on this evidence, is not yet a drop-in pipeline worker for this repo: competent Go, but its docs work needs a stricter reviewer arm (e.g. mixed roster: agy apply + codex/claude review) before its speed is usable.
One sub-agent per PR with an identical prompt: read the project constitution and quality/review policies, the intake and plan, and the full diff against the branch's own merge base; run go test + gofmt in the PR's own worktree; score six anchored dimensions 0–10 with mandatory file:line evidence; grade against the intake, never against sibling PRs. Anchors: 10 flawless · 8–9 minor nits · 6–7 one real gap · 4–5 significant gap · ≤3 major failure. The ttff scorers from the claude round onward carried the phrase-class + contrastive grep instruction; round-1 scorers reached comparable depth on their own initiative.
| Dimension | Weight |
|---|---|
| Intake fidelity — every "What Changes" item, nothing beyond scope | 20% |
| Correctness — logic read directly; tests run by the scorer | 20% |
| Diff hygiene — minimal diff, pattern conformance, commit/PR quality | 20% |
| Sweep completeness — mirror classes, aggregate specs, memory | 15% |
| Test quality — pin-by-value, coverage relocated not deleted | 15% |
| Prose & docs — present-truth framing, no transition narration | 10% |