- Entry date
- 15 August 2025
- Category
- Tooling
- Access
- 🔓 Public
We put an LLM inside the browser tab itself, via WebGPU. It worked for small models — and told us exactly where the ceiling was.
- WebLLM
- local, in-browser GPU inference via WebGPU
- OpenRouter
- cloud fallback for anything past the VRAM ceiling
- Monaco + Docker
- full IDE surface: editor, collaboration, containers
Honest evaluation
In-browser WebGPU inference genuinely worked for small models; for anything serious, the cloud fallback carried the real load — the central 'local-first' claim only partly holds.
What would prove or disprove it further
What would sharpen this further: tracking, across real usage, what fraction of requests actually stayed local versus fell back to the cloud. A local-completion rate would turn "the cloud did most of the real work" from an engineering impression into a measured number, and would show whether later consumer GPUs move the ceiling enough to matter.
The evidence — full reasoning behind the verdict
Verdict: partly-proven. In-browser, GPU-accelerated inference via WebGPU genuinely worked — a developer could run a real model against their own GPU with no server round-trip. But the central ambition, a local-first IDE, only partly holds: for anything past small models, the cloud fallback was doing the real work.
This landed here because the limit was measured, not guessed at: WebGPU exposes consumer VRAM directly, and that VRAM imposed a hard, observable ceiling on model size versus quality. The routing logic between local and cloud existed precisely because local alone wasn't enough — that's evidence the ceiling is real, not a design choice avoided out of caution.
Runnable proof — see it work
If a browser tab can run a model, does it need to? This project tested that directly — and found the honest limit rather than the hoped-for one.
What it was
A browser-based IDE — Monaco editor, live collaborative editing, Docker integration — with a genuinely novel twist: local, GPU-accelerated model inference running inside the browser itself via WebGPU, using WebLLM, with OpenRouter's cloud models as a fallback when local inference wasn't enough.
What we built
ResearchA working implementation across the full stack — React, Next.js, TypeScript, Monaco, WebLLM, Express.js, Docker, WebGPU. A developer could open the IDE, run a small model entirely client-side against their own GPU, and fall back to a cloud model through OpenRouter the moment the task outgrew what local inference could do.
The interesting engineering wasn't the editor — Monaco is a solved problem. It was the routing decision underneath: when to trust the browser's own GPU, and when to hand off.
What we learned — including the honest negative
WebGPU inference in-browser was real and it worked — but only for small models. Consumer-GPU VRAM, exposed through WebGPU rather than a native CUDA context, imposed a hard ceiling on model size versus quality that no amount of clever engineering moved. That ceiling is the actual finding here, and it's worth stating plainly: the cloud fallback was doing most of the real work for anything a developer would call "serious," not the local path.
That's not a failure of the idea — it's a measurement. Knowing exactly where the current limits of in-browser inference sit is more useful than a demo that quietly avoids testing them.
Where it went / status
A working implementation, not a maintained product. Its real value was diagnostic: this is the direct ancestor of the later, much more ambitious native (non-browser) GPU-operating-system inference work — the VRAM ceiling found here in a sandboxed browser context is the same ceiling that later work set out to push against directly, on real hardware, without a browser in the way.
What is still open — kept visible
The honest edges, next to the wins. This is what turns 🔬 into 🟢 — honestly.
- WebGPU inference was viable for small models only — a real VRAM ceiling on consumer GPUs, surfaced directly through the browser.
- The cloud fallback was doing most of the real work for anything serious; say so plainly rather than leading with the local-inference headline alone.
- A working implementation, not a maintained product — kept in the record for what it taught about the current limits, not as an ongoing tool.
Where this connects
Sources
- vocabotics project audit — AIDE / AI-Powered IDE (WebGPU local inference with cloud fallback), Aug 2025vocabotics internal project history · as of August 2025