VibeusLoadingЗагрузка
Vibeus

Loading is taking longer than usual.Загрузка занимает больше времени, чем обычно.

The full page couldn't load.Не удалось полностью загрузить страницу.

You can read the available content below or try loading the page again.Вы можете читать доступный контент ниже или повторить загрузку страницы.

How much does KV-cache quantization change a local model’s output?... - Vibeus

Owen Price ·

How much does KV-cache quantization change a local model’s output? A Qwen3.8 27B comparison tested 35 weight quantizations in llama.cpp, ExLlamaV3 and vLLM, with 16-bit, 8-bit and, where supported, 4-bit KV caches. The dataset combines agentic tool-use examples generated by the full-precision model with texts from different domains, at contexts up to 64k tokens.

For the tested llama.cpp and ExLlamaV3 configurations, an 8-bit cache roughly halves cache memory while changing KL divergence by less than 0.00012 in either direction. Some results moved slightly in the opposite direction, within numerical noise. vLLM’s FP8 cache added about 0.0004–0.0006 KL across the tested checkpoints; the author suggests that a change of attention backend may contribute, but did not establish that as the cause.

At 64k context the comparison lists 4.00 GiB for a 16-bit cache, approximately 2.13 GiB for llama.cpp q8_0 and EXL3 8-bit, and 2.00 GiB for vLLM FP8. A 4-bit cache uses about 1.13 GiB in the tested llama.cpp and EXL3 configurations, but has a measurable impact on the output distribution.

With llama.cpp’s q4_0 cache, the added KL was about 0.0016–0.0027 across the tested GGUF weights. The relative increase depended on the weight quantization: roughly 6–10% for the tested Q3-class weights, about 39% for UD-Q4_K_XL and about 390% for Q8_0, where KL rose from 0.0007 to 0.0034. The EXL3 4-bit penalty varied with weight precision, from about 0.0002–0.0004 at 2.0–2.5 bits per weight to about 0.0032–0.0033 at 6–8 bits per weight. The author does not recommend 4-bit KV cache for these tests.

Across 12 long documents, the extra 4-bit-cache KL was around 15–30% higher at positions 16k–32k than in the first 4k tokens. Only one document reached the 32k–64k range. Results beyond 64k were not measured; the author expects further degradation, while commenters questioned whether the trend can be extrapolated without additional tests.

The study also compared weight formats. When the author reran NVIDIA NVFP4 and Qwen FP8 weights in vLLM using Marlin kernels with BF16 activations, KL fell from 0.0256 to 0.0117 for NVFP4 and from 0.0039 to 0.0028 for FP8. The latter tied UD-Q5_K_XL in this comparison despite requiring 8.7 GiB more for loaded weights.

KL divergence here measures how far next-token probability distributions move from an unquantized BF16 reference; it is not a direct measurement of task success or generated-answer quality. The author used the top 200 log probabilities at each position to calculate a lower bound on full-vocabulary KL and paired KV-cache differences on identical tokens. Readers discussed the limits of that method and the absence of longer-context tests. The findings describe this model, dataset and configurations rather than a universal result for all local models.

Источник

Publication imagePublication imagePublication image