VibeusTwo LiveCodeBench v6 charts for DeepSeek-V4.1-Flash-1017 show how... - Vibeus
Two LiveCodeBench v6 charts for DeepSeek-V4.1-Flash-1017 show how much a benchmark result can move when the number of generations changes. In a comparison posted by u/Positive-Resource922, four Unsloth quantizations and the Q8 baseline all rose between 2.05 and 3.11 percentage points. IQ1_S went from 88.95% to 91.0%, IQ1_M from 88.76% to 91.2%, IQ2_XXS from 88.19% to 91.3%, Q2_K_XL from 89.33% to 91.7%, and Q8 from 88.95% to 91.7%. Three GSQ-RCO variants moved by −0.4, +0.7 and +1.9 points.
The original percentages are consistent with 525 outcomes across 175 problems, while the updated methodology reports 12 generations per problem, or 2,100 outcomes. If the original 525 are included in that total, the nine additional generations per problem would put the five Unsloth variants at roughly 91.6% to 92.6%. The comparison links an Unsloth team reply describing the earlier chart as one run followed by additional evaluations with unchanged settings.
The calculation depends on whether the initial outcomes were reused. Estimates of a joint probability also depend on independence, which is doubtful when related models answer the same problems. The charts alone cannot establish what produced the changes. In the discussion, readers asked for reproducible run details and pointed out that pass rates can miss other differences, such as whether an output follows a required JSON schema.
