VibeusvLLM nightly can now keep Qwen4Exp’s PLE n-gram table on disk... - Vibeus
vLLM nightly can now keep Qwen4Exp’s PLE n-gram table on disk instead of loading it into memory. PR #59958 is merged: enable disk_offload in the engram configuration and keep the checkpoint on local NVMe. Each step reads only the rows it needs.
On the contributor’s DGX Spark, Qwen3.8-Flash-Next NVFP4 loaded 72.09 GiB of weights instead of 99.09 GiB — 27 GiB saved. In that run, decode differed by at most 3.5%; an 8K prompt’s median time to first token rose from 2.54 to 2.82 seconds. GSM8K was 246/256 versus 247/256. The decode comparison used one run per configuration, so differences of a few percent are noisy.
BF16, FP8 and NVFP4 are supported. The OS can reclaim cached rows when memory is needed. Cold prefills cost more: roughly 1 GB read for an 8K prompt in the reported NVMe test, with about 20% higher first-token latency. Network storage was substantially slower.
This offloads the PLE table, not every model weight. The current implementation does not batch per-row reads for long prefills or support weight reload; Elastic EP is rejected.
