The sovereign inference appliance
In progress — the honest speed gap- VRAM vs. llama.cpp's 18.25 GB — 3.1× less, bit-exact
- 5.8 GB
VRAM vs. llama.cpp's 18.25 GB — 3.1× less, bit-exact- vs. llama.cpp's 201 — honestly ~24× slower, today
- 8.3 tok/s
vs. llama.cpp's 201 — honestly ~24× slower, today
A 30B-class MoE model, running on one consumer GPU, dequantized bit-exact against llama.cpp (0 mismatches across 114,688 values checked). The memory win is real and measured. The speed is not — we're not hiding that.
Fix planHonest update (re-measured 2026-07-10): the earlier FUSE and kernel-launch theories were both tested and falsified — FUSE overhead is ≈0% once warm, and batching the launches made it slower. The real limit is PCIe bandwidth: ~1.1 GB is streamed per token over a Gen3 link (12.4 GB/s), capping this box at ~10.7 tok/s. So the claim is the memory regime — 3.1× less VRAM, bit-exact — not the speed.
Source: efficient/DASHBOARD.md — Strand (E) Navigator, re-measured 2026-07-10 (W3, nav_moe_qwen3_gpudq_fast.cu)