VibeusA 68 GB Qwen3.8-Flash-Next quant on an RTX 3060 12 GB and 16 GB of... - Vibeus
A 68 GB Qwen3.8-Flash-Next quant on an RTX 3060 12 GB and 16 GB of single-channel DDR4: Betelgeuse streams MoE experts from NVMe instead of trying to keep the whole model in memory. Developer zyxciss reports roughly 22 tokens/s on repeated warm coding workloads, with brief peaks up to 48. The complete attached terminal demo shows the engine in use; it is not an independent reproduction of the benchmark.
This replaces an earlier Qwen3.6 prototype that discarded low-weight experts to inflate throughput and damaged output quality. Betelgeuse retains all 24,576 experts, executing the actual top 10 routed experts in each of 48 layers at the original quantized precision. The reported parity check covered 576 greedy output tokens and 28,080 router records across 17 configurations, with zero difference against the unmodified baseline. That is a bounded developer test, not proof of universal equivalence or full-precision model quality.
The test machine used CachyOS/Arch Linux, single-channel DDR4 at about 19 GB/s, a consumer PCIe NVMe drive around 2.1 GB/s, and a process limited to 12 GiB with swap disabled. In the developer’s stock mmap comparison, generation reached 1.47/1.63 tokens/s and physical reads 208.3/235.0 MiB per token. Strata reached 8.11 tokens/s using its low-RAM mmap path; the post contrasts that with 40+ tokens/s when 48–64 GB RAM can pin more than 24 GB of experts. These are hardware- and workload-specific measurements.
Dense layers, GDN, QSA and the LM head stay on the GPU, while about 5,000 VRAM expert slots are distributed unevenly: 240 per early layer, 92 in the middle and 60 near the end. Evicted experts move to a separate RAM victim cache. Remaining misses are fetched in aligned blobs through io_uring. An in-stream KICK/WAIT doorbell overlaps shared-expert computation with miss handling; staging per-layer embeddings on CUDA brings decode into one CUDA Graph replay instead of repeated per-layer CPU/GPU synchronization. The post estimates about 2 microseconds for resident GPU hits and roughly 360 ms/token lost to the old synchronization path. Those component timings are developer estimates. The optimized reader uses buffered I/O despite the historical “direct-io” flag name.
The upstream PR records two distinct benchmark pairs. Without automatic warmup, first/repeated decode was 16.29/22.03 tokens/s and prompt processing 13.85/19.35. With automatic warmup, decode was 16.80/21.85 and prompt processing 16.18/18.29. Physical reads fell from 35.37 to 12.43 MiB/token on the repeated request. The 16.80 result already followed about 42 seconds of startup warmup; combining it with 22.03 does not create a paired cold/warm test. Peaks of 48 are cache-hit stretches, not sustained throughput.
The current Alpha-Orionis-XS bundle reorganizes an existing ISTA GSQ-RCO-IQ2_XS quant; it is not a newly trained or distilled model. Removing duplicate expert payloads leaves four required files: a roughly 3.77 GB dense GGUF, 35.48 GB experts.bin, a 28.80 GB per-layer-embedding GGUF and a 6,920-byte hot-expert profile. Allow about 68 GB total. The profile supplies hints, not weights or a prewarmed cache. This custom stripped layout needs the Betelgeuse fork; it is not a standalone GGUF for stock llama.cpp, Ollama or LM Studio.
The current README explicitly calls the engine experimental and partly broken, recommends bel-server, and says the historical numbers predate the renamed, stripped bundle. That bundle has not received a new complete benchmark. The documented example targets Linux x86-64/CUDA sm_86, one server slot and a 512-token context; it does not validate full 262K context or promise that arbitrary models and prompts will run at 22 tokens/s. CLI configurations outside that envelope need separate validation, and no MTP setup is certified for exact-output paging.
The developer also reports 8/12 tokens/s cold/warm, with peaks of 21, for an approximately 190 GB Q8_K_XL build, versus out-of-memory failures or below 0.1 tokens/s in stock llama.cpp. A DeepSeek V4.1 Flash port is still being developed, with an early claim around 25 tokens/s. Neither claim is an independently reproduced result here. Code, model bundle and PR links are in the original post. The llama.cpp PR remains an open RFC/reference implementation, not a merged upstream feature. Engine code is MIT; the base weights retain Qwen Community License 1.0 and its separate commercial-use restrictions.