How to Run AI Models Directly in the Browser with Transformers.js and WebGPU
- 2 hours ago
- 13 min read
Introduction
Every time you reach for an AI feature in your app, the same wall appears: you need an API key, a running server, a billing account, and a promise that your users' data will be handled responsibly somewhere in a cloud you don't control. For indie developers, privacy-focused startups, and CS students building weekend projects, those constraints aren't just inconvenient — they're dealbreakers.
What if the model ran entirely on the user's device, inside the browser, with no backend at all?
That's exactly what browser-native AI makes possible. Using WebGPU, ONNX Runtime Web, and the Transformers.js library from Hugging Face, you can load a quantised language or vision model, run a full inference pass, and return output to the user — all without a single network call to a server you own.
Real-world applications you can build this way:
Offline-capable AI assistants that work without internet connectivity
Privacy-first apps where user data never leaves the device
Edge and IoT dashboards with on-device vision or NLP inference
Student and hobbyist projects with no API budget
AI feature prototypes before committing to cloud infrastructure
Browser extensions that classify, summarise, or transform content locally
When you are ready to go from architecture diagram to a deployed, production-grade feature, contact our team at build.codersarts.com/contact or drop us an email at contact@codersarts.com to discuss how we can build it for you.
How It Works: Core Concept
Why the Obvious Approach Fails
The obvious approach to adding AI to a web app is to call an API. You send a request to OpenAI or Anthropic, they run the model on their GPU cluster, and you stream the response back. This works fine until you hit any of the following: a user who is offline, a compliance requirement that prohibits sending personal data to third parties, an API bill that scales faster than your revenue, or a latency budget that a round-trip to the cloud can't meet.
The next obvious answer is to run your own model server. Now you have infrastructure costs, a deployment pipeline, cold-start latency, and still the same data-egress problem.
The Browser-Native Answer
Browser-native AI takes a different path entirely. The model file is downloaded once, cached in the browser's Cache API, and then all inference happens locally using the GPU (via WebGPU) or CPU (via WebAssembly). The server's only job is to serve a static HTML/JS bundle and, on first load, the model weights. After that, it can go away entirely — the app works offline.
Think of it like the difference between streaming a movie (server required every time) and downloading it to your device (server needed once, then you own a local copy). The model is the movie. Once it's cached, you don't need the studio's servers to watch it again.
Data-Flow Diagram
SETUP PHASE (first load only)
──────────────────────────────────────────────────────────────────
Browser CDN / Hugging Face Hub
│ │
│ GET static assets │
│ ─────────────────────── ► │
│ ◄─────────────────────── │
│ │
│ GET quantised model │
│ (.onnx / ~100 MB) │
│ ─────────────────────── ► │
│ ◄─────────────────────── │
│ Store in Cache API │
│ (localStorage not used) │
▼
[Model cached on device]
RUNTIME PHASE (every subsequent inference)
──────────────────────────────────────────────────────────────────
Main Thread Web Worker GPU / WASM
│ │ │
│ Post { input } │ │
│ ──────────────────────► │ │
│ │ Load model from cache │
│ │ (first inference only)│
│ │ ─────────────────────► │
│ │ Forward pass │
│ │ ◄───────────────────── │
│ │ Stream tokens / result│
│ ◄────────────────────── │ │
│ Render output │ │
▼ ▼ ▼
UI updated Worker idle GPU/WASM idle
(no server call at any stage)
The critical insight: once the model is cached, the entire pipeline runs in the right half of this diagram — no arrows cross the network.
System Architecture Deep Dive
Architecture Overview
A browser-native AI application has no traditional backend. Instead, it has four layers that all live in the client:
Presentation Layer — A React (or plain JS) UI that accepts user input (text, image, file), displays streaming output, and manages loading/error states. This is what the user touches.
Orchestration Layer — The main thread, which coordinates input handling, spawns Web Workers, passes messages to the inference layer, and renders responses. It never runs the model directly.
Inference Layer — A Web Worker running Transformers.js or ONNX Runtime Web. This is where the model lives and where the forward pass executes. Moving inference off the main thread is non-negotiable; a 200 ms forward pass on the main thread freezes the entire UI.
Acceleration Layer — WebGPU (GPU acceleration, supported in Chrome 113+ and Edge 113+) with an automatic fallback to WebAssembly (WASM) for browsers where WebGPU is not available. Transformers.js handles the backend selection transparently.
Model Storage Layer — The browser's Cache API, used to store the model file after the first download. Cache API entries persist across sessions, so a 150 MB model downloaded once stays available indefinitely.
Component Table
Component | Role | Options |
UI Framework | Render input/output interface | React, Vue, Svelte, plain JS |
Build Tool | Bundle and serve the app | Vite, Webpack, Parcel |
Inference Library | Load model, run forward pass | Transformers.js (HF), ONNX Runtime Web |
Acceleration Backend | GPU or CPU inference | WebGPU, WASM (auto-detected) |
Web Worker | Off-thread inference execution | Native Web Worker, Comlink |
Model Format | Serialised model weights | ONNX (quantised INT4/INT8) |
Model Source | Download and serve weights | Hugging Face Hub, self-hosted CDN |
Caching Layer | Persist model between sessions | Cache API (Service Worker optional) |
Cross-Origin Headers | Enable SharedArrayBuffer | COOP + COEP server headers |
Streaming Layer | Return tokens incrementally | postMessage loop, ReadableStream |
Data Flow Walkthrough
User opens the app. The browser fetches the static HTML/JS/CSS bundle from a CDN.
App initialises. The main thread checks the Cache API for the model file.
Cache miss (first load). The model is fetched from Hugging Face Hub (or your CDN), stored in Cache API, and a progress bar is shown.
Cache hit (subsequent loads). The model is read from cache — no network request.
User submits input. The main thread validates the input and posts a message to the Web Worker: { type: "RUN_INFERENCE", payload: { text: "..." } }.
Worker receives the message. If the model is not yet loaded into the worker's memory, it loads it now from the Cache API. This adds ~1–3 seconds on the first inference per session.
Forward pass executes. Transformers.js (or ONNX Runtime Web) runs the model on WebGPU or WASM, generating output tokens or a classification result.
Worker streams results. For generative models, the worker posts each token back to the main thread as it is produced.
Main thread renders. React state updates on each token message, streaming text to the UI in real time.
Inference completes. The worker sends a { type: "DONE" } message. The main thread re-enables the input field.
Non-Obvious Design Decisions
Why Cache API instead of IndexedDB for the model?
Model files are binary blobs, sometimes 200–500 MB. The Cache API is designed for binary response caching, supports range requests, and integrates cleanly with Service Workers for offline-first patterns. IndexedDB can store blobs too, but its API is significantly more complex for this use case and its performance with large binary objects varies across browsers.
Why a dedicated Web Worker instead of just async/await on the main thread?
JavaScript's event loop is single-threaded. Even with await, a computationally intensive WASM or WebGPU operation blocks the main thread's ability to process UI events. A Web Worker runs in a completely separate thread, meaning your loading spinner keeps spinning and your cancel button keeps responding even during a 500 ms forward pass.
Tech Stack Recommendation
Choosing the right stack depends on your timeline and scale requirements. Here are two opinionated recommendations.
Stack A — Beginner / Prototype (Build in a Weekend)
This stack minimises configuration. You can have inference running in a browser in under two hours.
Layer | Technology | Why |
UI | Plain HTML + Vanilla JS | Zero build setup, instant feedback |
Bundler | Vite | One command (npm create vite@latest), fast HMR |
Inference library | Transformers.js | Abstracts WebGPU/WASM selection, huge model variety |
Model | Hugging Face Hub (CDN) | Free, no self-hosting required |
Caching | Cache API (manual fetch) | Built into the browser, no dependencies |
Deployment | Netlify / Vercel (free tier) | Supports custom COOP/COEP headers via config file |
Cross-origin headers | netlify.toml / vercel.json | Required for SharedArrayBuffer |
Estimated monthly cost: $0 (Vercel/Netlify free tier, Hugging Face Hub free model downloads under rate limits)
Stack B — Production-Ready (Designed to Scale)
Layer | Technology | Why |
UI Framework | React 18 + TypeScript | Type safety, component reuse, streaming hooks |
Bundler | Vite + custom COEP plugin | Fine-grained header control per route |
Inference library | ONNX Runtime Web | Lower-level, more control over quantisation and execution providers |
Worker abstraction | Comlink | Removes boilerplate from postMessage / onmessage wiring |
Model hosting | Self-hosted S3 + CloudFront | Predictable latency, no HF rate limits, model versioning |
Service Worker | Workbox | Precaches model on install, enables true offline mode |
Monitoring | Sentry + custom Web Vitals | Track inference latency and WASM fallback rates |
Deployment | AWS CloudFront + S3 | Custom headers, edge caching, SLA guarantees |
CI/CD | GitHub Actions | Automated build + deploy on push |
Testing | Playwright + Vitest | Unit tests for inference logic, E2E for UI flows |
Estimated monthly cost: ~$5–$20/month for CloudFront + S3 (depends on model size and traffic volume)
Implementation Phases
Building a browser-native AI app is a four-phase process. Each phase has clear entry and exit criteria, and each introduces a distinct category of technical decision.
Phase 1: Environment Setup and Cross-Origin Configuration
What you are building: A working Vite project with the correct HTTP response headers, a basic React shell, and a verified Transformers.js installation.
Key technical decisions:
Which hosting provider to use, and how to configure COOP (Cross-Origin-Opener-Policy: same-origin) and COEP (Cross-Origin-Embedder-Policy: require-corp) headers. These headers are mandatory for SharedArrayBuffer (required by ONNX Runtime Web's WASM backend) and for WebGPU on some platforms. Misconfiguring them silently breaks third-party iframes, OAuth popups, and some analytics scripts.
Whether to use a Service Worker for header injection (works on any static host) or rely on server/CDN-level header configuration.
TypeScript vs. plain JS — the Transformers.js types are comprehensive and worth adopting from the start.
If you're struggling to configure these headers without locking out your critical third-party dependencies, our team can audit and implement this safely for you. Get in touch at build.codersarts.com/contact
Phase 2: Model Selection, Download, and Caching
What you are building: A model loader that fetches a quantised ONNX model from Hugging Face Hub on first load, stores it in the Cache API, and displays a progress bar. On subsequent loads, it reads the model from cache with no network request.
Key technical decisions:
Which model to use. Transformers.js supports hundreds of models, but not all of them are quantised for browser use. You need an ONNX-exported, INT4 or INT8 quantised variant. Common good starting points: Xenova/distilbert-base-uncased-finetuned-sst-2-english for classification, Xenova/whisper-tiny.en for speech-to-text, Xenova/gpt2 for generation.
INT4 vs. INT8 quantisation. INT4 produces smaller files (sometimes 2–4× smaller than INT8) but can degrade accuracy meaningfully on tasks requiring nuanced reasoning. INT8 is a safer default for most applications.
How to handle the model download progress event and translate it into a UI progress bar — Transformers.js fires progress callbacks that need to be relayed from the worker to the main thread via postMessage.
Need help selecting and benchmarking the perfect model size for your specific business requirements? Let our team analyze the accuracy trade-offs and build it for you. Reach out at contact@codersarts.com
Phase 3: Web Worker Architecture and Message Passing
What you are building: A Web Worker that encapsulates all inference logic, receives input messages from the main thread, runs the forward pass, and streams results back. The main thread never touches the model directly.
Key technical decisions:
Worker communication protocol: raw postMessage / onmessage is simple but error-prone as complexity grows. Comlink (a thin wrapper from Google) lets you call worker functions as if they were async functions on the main thread, eliminating most of the message-type boilerplate.
How to handle worker lifecycle: should the worker be created once on app startup and reused, or created per-request? Creating it once saves the ~100–300 ms instantiation cost on every query, but requires careful state management if the model needs to be swapped.
Error propagation: if the worker crashes or the model throws, the error needs to be caught inside the worker and posted back to the main thread explicitly — uncaught worker errors do not bubble to the main thread's window.onerror.
Setting up a robust communication protocol between threads can quickly turn into a development bottleneck. If you want a battle-tested worker architecture tailored to your app, let’s talk strategy at build.codersarts.com/contact
Phase 4: Streaming Output and UI Integration
What you are building: A real-time streaming interface that displays model output token by token as it is generated, with a cancel button, a latency readout, and a graceful WASM fallback indicator.
Key technical decisions:
Token streaming without Server-Sent Events: in a server-based LLM setup you'd use SSE or WebSocket streaming. In the browser-native setup, the worker posts each generated token via postMessage, and the main thread appends it to a React state variable using a functional update. The streaming loop is custom — there is no framework magic here.
WebGPU availability detection: navigator.gpu is undefined in Firefox and in some older Chromium builds. You need a runtime check before calling pipeline(..., { device: "webgpu" }). If WebGPU is unavailable, fall back to device: "wasm" and display a warning to the user explaining that inference will be slower.
Rendering performance: updating React state on every token (potentially 20–50 times per second) can cause jank if done naively. Batching token updates using useRef + a requestAnimationFrame flush loop keeps the UI smooth.
To eliminate UI stuttering and implement a flawless requestAnimationFrame batching loop, partner with our engineering team directly by emailing contact@codersarts.com.
Phase 5: Deployment and Production Hardening
What you are building: A production build deployed to a CDN with correct headers, a self-hosted model on S3/CloudFront, a Service Worker for offline support, and basic error monitoring.
Key technical decisions:
Whether to host the model on Hugging Face Hub (free, but subject to rate limits and Hub availability) or self-host on S3 + CloudFront (costs money, but gives you SLA guarantees and version control over model files).
Content Security Policy (CSP) configuration: blob: and worker-src directives need to be explicitly allowed for Web Workers to function. Many default CSP configurations block this.
Service Worker scope: if your app is deployed under a subpath (e.g., example.com/app/), the Service Worker's scope must match — a common source of confusing offline failures.
For a frictionless, secure deployment that hits all your enterprise security requirements, let our cloud and AI engineers manage the integration. Drop us a line at build.codersarts.com/contact.
Common Challenges
Every developer building a browser-native AI app hits the same walls. Here are the non-obvious ones, and what actually fixes them.
1. WebGPU Is Not Available — But the Error Is Silent
Problem: navigator.gpu exists on the Window object, so your feature-detection check passes. But calling requestAdapter() returns null because the user's GPU driver doesn't support the required Vulkan/Metal/D3D12 backend.
Root cause: WebGPU availability is a two-step check: the API must exist and a compatible adapter must be available. Many headless environments, VMs, and older integrated GPUs pass the first check and fail the second.
Fix: Always await navigator.gpu.requestAdapter() and check for null before proceeding. Fall back to WASM automatically and log the fallback for monitoring.
2. The Model Re-Downloads on Every Visit
Problem: Users report a 2–5 minute loading screen on every visit, even though you're using the Cache API.
Root cause: The model URL includes a version hash that changes when you update the model. The cache key no longer matches, so the browser treats it as a new resource.
Fix: Adopt a stable model URL strategy — either pin to a specific commit hash on Hugging Face Hub (not main) or self-host the model at a versioned URL you control. Implement a cache cleanup routine to remove stale model versions.
3. The UI Freezes During Inference
Problem: Despite async/await everywhere, the app becomes unresponsive for 200–800 ms during each forward pass.
Root cause: WASM execution is synchronous and runs on the main thread if the model is loaded directly in the page (not in a Worker). Even if the load is await-ed, the computation blocks the event loop.
Fix: Move all model loading and inference to a Web Worker. No exceptions.
4. SharedArrayBuffer Is Blocked
Problem: ONNX Runtime Web crashes with SharedArrayBuffer is not defined in production but works locally.
Root cause: SharedArrayBuffer requires cross-origin isolation, which requires Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp headers. Vercel, Netlify, and GitHub Pages do not set these by default.
Fix: Add the headers explicitly in your deployment configuration (vercel.json, netlify.toml, or a Service Worker that intercepts responses and injects the headers).
5. Quantised Model Produces Garbage Output
Problem: The INT4 model you chose produces fluent-looking but factually wrong or incoherent text.
Root cause: INT4 quantisation aggressively reduces precision. Some model architectures tolerate it well; others do not. This is task- and model-specific and cannot be predicted without benchmarking.
Fix: Always benchmark INT4 against INT8 on a representative sample of your task's inputs before shipping. If INT4 quality is unacceptable, move to INT8 (larger file, better accuracy) or use a different model family.
6. Token Streaming Causes UI Jank
Problem: The text output stutters and the frame rate drops below 30 fps during generation.
Root cause: Posting and processing a postMessage for every single token, and calling a React state setter on each one, creates too many microtask queue flushes per second.
Fix: Batch tokens in the worker (e.g., flush every 3–5 tokens or every 16 ms), and on the main thread use a useRef buffer with a requestAnimationFrame loop to drain it into React state once per frame.
7. COEP Breaks Third-Party Embeds
Problem: After enabling COOP/COEP headers, your Stripe payment iframe, Google Maps embed, or HubSpot chat widget stops loading.
Root cause: COEP requires all subresources (including iframes) to opt in to cross-origin isolation via the Cross-Origin-Resource-Policy header. Third-party services often do not set this header.
Fix: Evaluate which third-party embeds you actually need on the same page as inference. Isolate the AI feature to a dedicated route or iframe where COEP headers apply only to that page. Alternatively, use a Service Worker to inject headers selectively.
Solving these edge cases took us over 60 hours of testing across different browsers, operating systems, and GPU configurations. Save your development team the headache and let us implement these proven fixes directly into your codebase. Reach out to us today at contact@codersarts.com.
Ready to Build This Yourself?
Understanding the architecture is one thing, but engineering and deploying a production-ready, browser-native AI feature tailored to your specific product constraints is another.
Whether you need help optimizing your WebGPU/WASM fallbacks, structuring complex Web Worker pipelines, or safely managing asset delivery via self-hosted CDNs, the Codersarts team can help you bypass the edge-case headaches and ship faster.
How we can partner with you:
Custom Engineering & Architecture: Full, end-to-end development of your client-side AI features.
Performance & Token Optimization: Expert tuning of model quantisation (INT4 vs INT8) and UI streaming batch loops.
Infrastructure & Header Configuration: Seamless implementation of COOP/COEP isolation headers without breaking your third-party embeds or auth flows.
🚀 Let's Build It Together Tell us about your project requirements and let's bring your browser-native AI features to life.
🌐 Get in Touch: build.codersarts.com/contact
📧 Email Us Directly: contact@codersarts.com
Conclusion
Browser-native AI is not a toy. It is a legitimate architectural pattern for any application where privacy, offline capability, cost, or latency make server-side inference impractical. The stack — Transformers.js, ONNX Runtime Web, WebGPU, Web Workers, and the Cache API — is production-ready today, and the ecosystem is improving rapidly.
If you are starting from scratch, begin with Stack A: Vite + Transformers.js + Hugging Face Hub + Vercel. You can have a working prototype running inference in the browser in a single afternoon. Once you understand the model caching and Web Worker patterns, moving to a production architecture is straightforward.
When you are ready to go from architecture diagram to deployed app, contact us at contact@codersarts.com
Comments