colibri
An open-source pure-C inference engine for running frontier Mixture-of-Experts models on hardware you already own, streaming routed experts across VRAM, RAM, and NVMe.
colibri is an open-source inference engine and research platform for running large Mixture-of-Experts models on consumer and heterogeneous hardware. It treats VRAM, RAM, and storage as one placement hierarchy: fast-memory limits can reduce speed, but the default policy does not silently change model precision or router semantics. The repository currently supports nine model families, from 744B-parameter GLM models to 2.8T-parameter Kimi K3, plus GLM-5.3-Flash with vision, DeepSeek V4 Flash and V4.1 Flash, Qwen3.8-Flash-Next, Qwen3.6, and OLMoE. The engine is a single C file with small headers, has no BLAS or Python runtime dependency, and does not require a GPU. Use ./coli chat for terminal inference, ./coli serve for a service endpoint, or ./coli web for the browser dashboard. The dashboard exposes live token metrics, a per-turn time breakdown, the VRAM/RAM/disk tier bar, a Brain page showing routed experts, and an Atlas page with the measured expert map. The runtime also includes model streaming, routing-aware caching and prefetch, CPU/CUDA/Metal execution, optional multi-SSD storage, and a local cluster mode. The project is intended for developers and researchers who want to inspect and improve the inference path rather than rent access behind an API. Speed depends heavily on the host, storage, memory, model, and workload; the README explicitly treats performance ideas as hypotheses until controlled end-to-end measurements support them.