VibeusAn LLM inside an LLM: QwenPU runs stories260K in Qwen3.5-4B’s... - Vibeus
An LLM inside an LLM: QwenPU runs stories260K in Qwen3.5-4B’s forward passes.
Divyansh Agrawal has taken his Qwenputer experiment beyond a RISC-V CPU: a compiled vector unit now runs Andrej Karpathy’s 260,000-parameter story model as a C program inside Qwen. Matrix multiplication, softmax and attention execute in the model’s forward passes, with the small model’s weights held in its internal memory rather than sent to an external tool.
The added circuitry is designed by a compiler, rather than learned during training. Qwen3.5-4B’s original weights stay unchanged; 13 computer layers are placed before its 32 layers. The story model’s int4 weights are loaded as program data into the vector unit’s memory, rather than baked into the checkpoint.
Agrawal’s repository reports about 53,000 trace tokens for each story token, down from 225,000 with the previous Qwenputer NPU and 2.43 million for the plain-C int8 version. The computer-layer GPU executor reaches roughly 3,300 trace tokens per second on an L4: around 16 seconds per story token, versus 41 previously. The attached 16-second clip is an accelerated visualization of that process, not real-time generation.
For correctness, the author reports checking all 3,184,920 story-program steps on the GPU executor against a reference, while the checkpoint-layer test covers the first 30,000 steps. These are the developer’s results; the model has not been run independently here.
The complete bf16 checkpoint is still 26.7 GB, much of it padding, and needs a GPU with at least 40 GB to load whole. The computer-only demo and checks fit a 24 GB GPU. A detailed write-up, further optimization and size reduction are still in progress. Code is MIT-licensed; model weights use Apache-2.0.