fab-kit findings · 2026-08-10 · experiment report
Two draft intakes — rvza (config-show-compose-defaults, a surgical Go fix) and ttff (agy-interactive-pane-capability, a prose-heavy repo-wide sweep) — were each run to a draft PR twice through the full fab-fff pipeline: once with every stage worker on kimi (kimi --auto, k3) and once on codex (gpt-5.6-sol @ xhigh for apply/review/review-pr). All four worktrees were created in the same 8-second window and ran concurrently.
Verdict: kimi was ~2.5× faster on both changes; codex shipped the higher-quality PR on both.
On the small surgical change the quality gap is marginal (8.65 vs 8.40) and kimi's speed wins the economics. On the sweep-heavy change the gap is decisive (9.45 vs 8.65): kimi left two stale normative spec claims that the plan's token-based verification grep could not catch, while codex's sweep survived an adversarial re-grep intact.
Same intake, same pipeline, same launch instant per pair — only the worker provider differs. All four PRs are draft, CI-green, and were produced without human intervention.
| PR | Arm | Change | Worker roster | Wall-clock | Rework cycles | Copilot comments | CI |
|---|---|---|---|---|---|---|---|
| #570 | kimi | rvza | kimi --auto (k3), all stages | 13m 35s | 0 | 0 | pass |
| #571 | kimi | ttff | kimi --auto (k3), all stages | 28m 04s | 1 | 5 (all fixed) | pass |
| #572 | codex | rvza | sol@xhigh; hydrate sol@high; ship luna@low | 33m 27s | 0 | 1 (fixed) | pass |
| #573 | codex | ttff | sol@xhigh; hydrate sol@high; ship luna@low | 71m 42s | 2 | — (review-pr not yet run) | pass |
Stage spans reconstructed from .fab-dispatch/ record and result-file timestamps; x-axis is minutes since worktree creation. Gaps between segments are orchestrator triage time. Only the final rework cycle's files survive, so the apply segment of a reworked run includes its interleaved failed reviews. The vertical PR tick is the measurement endpoint; the dashed gray segment after it is the post-PR Copilot review-pr stage, outside the wall-clock number.
| Run | Worktree born | Apply span | Review | Hydrate | Ship / PR | Review-pr |
|---|---|---|---|---|---|---|
| #570 kimi·rvza | 11:24:57 | 11:27:50–11:31:43 | 11:32:44–11:34:14 | 11:35:05–11:36:06 | PR 11:38:32 | 11:39:44–11:44:59 |
| #571 kimi·ttff | 11:25:03 | 11:28:15–11:45:46 (×1 rw) | 11:46:16–11:48:01 | 11:48:59–11:50:29 | PR 11:53:07 | 11:54:13–12:03:36 |
| #572 codex·rvza | 11:25:00 | 11:28:46–11:43:57 | 11:45:06–11:50:37 | 11:52:16–11:55:44 | PR 11:58:27 | 12:04:23–12:13:30 |
| #573 codex·ttff | 11:25:05 | 11:28:02–12:19:29 (×2 rw) | 12:20:10–12:27:38 | 12:28:22–12:32:53 | PR 12:36:47 | not yet run |
Each PR was scored blind-to-its-sibling by an independent sub-agent against the same anchored 0–10 rubric, grading the final branch state against the intake. Every score is backed by cited file:line evidence, and each scorer ran the Go test suite in its PR's worktree itself. Weights: intake fidelity 20% · correctness 20% · diff hygiene 20% · sweep completeness 15% · test quality 15% · prose & docs 10%.
Top strengths and weaknesses per PR, as reported by each scoring sub-agent (full evidence in the scoring transcripts).
"The implementation is the right one … precedence, empty-leaf fall-through, and the single-load warning rule are inherited rather than re-derived. The one real gap is test coverage of the empty-repo case … minor hygiene noise keeps this short of the top band."
"A tight, well-targeted change that does exactly what the intake specified and nothing more … What keeps it out of the top band is self-review rather than judgment."
"Technically … clean and well-shaped. It falls short on exactly the axis the intake flagged as the repo's #1 rework cause … Fixing the two lines and widening the sweep grep to phrase classes would take this to a 9-plus."
"A near-exemplary execution of a change that is 80% sweep discipline and 20% code … What keeps it off a perfect score is cosmetic."
Kimi's apply on the surgical rvza change took 3m 53s to codex's 15m 11s for near-equivalent output (8.40 vs 8.65). Review, hydrate, and ship stages show the same ~3× per-stage ratio. The orchestrator overhead between stages is comparable across arms.
On the small Go fix, kimi delivered 97% of codex's quality score at 2.5× the speed — the clear pick. On the sweep-heavy prose change, codex's xhigh-effort diligence bought the one thing that matters in this repo (sweep 10 vs 6) — the two claims kimi missed are exactly the class that historically comes back as rework, which costs more than the 44 extra minutes.
Both arms ran the same plan with the same verification grep tokens. Codex re-derived the sweep class beyond the tokens; kimi trusted the tokens. Sweep-verification greps should target phrase classes of the retired claim ("headless grammar only", "carries a headless_command only"), not just its canonical vocabulary.
Codex's ttff run was sent back twice by its own (sol) reviewer and emerged with the highest score of the four and a clean internal review (0 findings). Kimi's pipeline accepted after one rework — and Copilot then raised 5 comments on the PR. The internal review gate's strictness tracks the worker model's, which compounds the quality gap.
Both providers drove every stage — including pane-worker delivery, rework choreography, ship, and (3 of 4 so far) Copilot-comment triage — to CI-green draft PRs unattended. The provider choice is now a pure cost/quality/speed knob.
One sub-agent per PR, launched in parallel with an identical prompt: read the project constitution and quality/review policies, the intake and plan, and the full main...HEAD diff; run go test + gofmt in the PR's own worktree; score six anchored dimensions 0–10 with mandatory file:line evidence; grade against the intake, never against the sibling PR. Anchors: 10 flawless · 8–9 minor nits · 6–7 one real gap · 4–5 significant gap · ≤3 major failure.
| Dimension | Weight |
|---|---|
| Intake fidelity — every "What Changes" item, nothing beyond scope | 20% |
| Correctness — logic read directly; tests run by the scorer | 20% |
| Diff hygiene — minimal diff, pattern conformance, commit/PR quality | 20% |
| Sweep completeness — mirror classes, aggregate specs, memory | 15% |
| Test quality — pin-by-value, coverage relocated not deleted | 15% |
| Prose & docs — present-truth framing, no transition narration | 10% |