VibeusFaster token generation did not necessarily mean a faster bug fix... - Vibeus
Faster token generation did not necessarily mean a faster bug fix in u/FeydRowan’s private C++/CUDA experiment. The 3-bit Qwen3.8-27B decoded at 153 tok/s versus 132 for NVFP4, yet needed 18 minutes per task instead of 13. Its score was similar; the extra work appeared in longer reasoning.
The setup: nine real regressions from capy-flash, the author’s own inference engine, restored to the commit before each fix. Reports supplied symptoms, numbers and logs, without pointing to the faulty code. Cases included performance cliffs, numerical drift and a failure specific to WSL2 with two GPUs. These tasks had never been public. Local inference used capy-flash on two RTX 5070 Ti 16 GB cards, a Ryzen 9900X and 96 GB DDR5; Flash-Next used no expert skipping.
Each agent had to diagnose the cause and patch the code, with an 80-step limit and three runs per task/model. Its sandbox was on a Mac without network access or a CUDA toolchain: the agent could not build or run the code. Claude models used Claude Code; local models used OpenCode. A bridge that put the local models into Claude Code had performed much worse, so the author used each side’s better harness.
Every patch received two scores. An automatic hidden key checked the files/functions, patch precision and selected code conditions against the real fix. Claude Opus 5 also judged blindly from the report, code, compiler output and the real fix, described as one acceptable solution rather than the only solution. Its grades were 1, 0.75, 0.5, 0.25 or 0; a non-compiling patch was capped at 0.25 in both methods.
Per-patch correlation between the scores was r = 0.79, with the same model order. The judge was slightly stricter: judge minus automatic score was negative for every Claude model and near zero for Qwen. That agreement is the author’s check against family bias, not an independent replication. Preparing fair hidden keys is slow, and one model’s 27 attempts took 4–8 hours on the home machine. Confidence intervals were roughly ±0.2; differences under about 0.1 were too close to rank confidently.
Results below are judge score / tokens per task including reasoning / minutes per task. Claude rows used the API; Qwen and Gemma were local. The original chart also contains automatic scores, 95% confidence intervals and compilation failures.
Claude Opus 5.5: 0.74 / 16k / 3
Claude Sonnet 5.5: 0.58 / 8k / 1
Qwen3.8-Flash-Next NVFP4: 0.50 / 53k / 10
Qwen3.8-Flash-Next 3-bit: 0.48 / 53k / 9
Claude Haiku 5.5: 0.47 / 45k / 4
Qwen3.8-27B 3-bit: 0.39 / 111k / 18
Qwen3.8-27B NVFP4: 0.34 / 81k / 13
Claude Sonnet 5: 0.28 / 43k / 8
Claude Sonnet 4.5: 0.19 / 15k / 6
Claude Haiku 4.5: 0.13 / 23k / 4
Gemma 4 26B-A4B, thinking: 0.07 / 33k / 5
The paired quantization comparison is the key point. Flash-Next, with only its experts at 3 bits, lost 0.02 score and used ×1.03 tokens: little measurable change on these nine tasks. The dense 27B gained 0.05 score, within the noise, but produced 40% more tokens, ×1.40 with CI 1.25–1.59. Part of its elapsed-time gap came from a slow 3-bit prefill path that the author later fixed; a rerun on the newer build still used ×1.30 tokens. A higher tok/s figure therefore did not make that configuration a better deal per bug fixed.
Compilation failures did not establish a clear quantization trend: Flash-Next had 5/27 at 3-bit versus 1/27 at NVFP4; the dense 27B had 2/27 versus 4/27. In this setup Flash-Next clustered with Haiku 5.5 and above the older Sonnets, while Sonnet 5.5 and Opus 5.5 scored higher. Sonnet 5.5 used 8k tokens versus Flash-Next’s 53k; the prose describes roughly 1.5 versus 10 minutes, while the table rounds Sonnet to 1 minute.
The MoE exceeded the dense 27B by a paired 0.16, with the author reporting P = 0.92, and used fewer tokens. Gemma 4 26B did poorly as an agent here: with thinking disabled it never edited a file across nine tasks; with thinking enabled it edited, but mostly incorrectly.
The limits matter: nine tasks from one GPU-inference codebase, reports written in Italian, different agent harnesses and a Claude-family judge. This measures diagnosis and patching from code reading, not the full edit-build-test cycle or a general model ranking. The tasks remain private to preserve their usefulness, so readers cannot reproduce the benchmark from this post. The author asks whether others testing quantized models on real work see the same pattern: similar scores, but more thinking.
