VibeusLFM3-1.7B at 256k context fits in about 3.6 GiB of VRAM in... - Vibeus
LFM3-1.7B at 256k context fits in about 3.6 GiB of VRAM in u/crusaderky’s quantization report. That includes roughly 1.4 GiB for weights, 1.1 GiB for the context cache and 1.1 GiB for scratch buffers. The report found no discernible degradation on its measurements with Bartowski Q6_K_S weights and a q6_0/q5_0 key/value cache. This is a measured result for the tested setup, rather than a guarantee for every task.
A tighter 2.2 GiB configuration uses Q4_K_M, a 128k context and BeeLLaMA.cpp with q4_0/q3_0 cache and a 1,024-token tail. The author reports some quality loss and roughly threefold slower prefill on an RTX 3090 for the tail approach. The KLD tests do not cover the worst case of retrieving important information from far back in the context. Pushing compression further produced a sharp quality drop in the report.
The graphs compare mean and tail KL divergence, top-token agreement, sampled-token agreement and perplexity. The lighter LiquidAI QAD-Q4_0 variant did worse on these measures than Bartowski’s Q4_K_S. The comparison with MiniCPM5-2B comes from LiquidAI’s own benchmarks, separately from this quantization test.
