A developer known as slvDev has demonstrated a 28.9-million-parameter language model generating text at 9.88 tokens per second on an ESP32-S3 microcontroller that costs less than $10. The project, published on GitHub under an MIT license in late July 2026, fits the model onto a chip with only 512 KB of SRAM and 8 MB of PSRAM by keeping the majority of its parameters in flash storage rather than working memory. The approach borrows Per-Layer Embeddings from Google's Gemma 3n architecture, allowing roughly 25 million parameters to reside in 16 MB of flash while only the active rows — about 450 bytes per token — are pulled into faster memory during inference.
The result is a fully offline language model that runs entirely on the device with no network connectivity, displaying generated text on a small wired screen. While the model is limited to writing short, coherent stories in the style of the TinyStories dataset — it cannot answer questions, follow instructions, write code, or retrieve facts — the achievement highlights how memory-hierarchy-aware architectures can bring generative AI to the smallest and cheapest embedded platforms. The repository has gathered over 3,600 stars and more than 470 forks since its release, and a second model named Barista demonstrates the same technique for a narrow espresso-question-answering task at roughly twice the token rate.
What's New / Specs
- Model: 28.9M parameters trained on Microsoft Research's TinyStories dataset (arXiv:2305.07759)
- Hardware: ESP32-S3 with 512 KB SRAM, 8 MB PSRAM, 16 MB flash; unit cost under $10
- Quantization: 4-bit precision, reducing weight memory from ~60 MB (16-bit) to 14.9 MB
- Memory layout: ~25M parameters (12 MB at 4-bit) in flash via Per-Layer Embeddings; dense core and output head in PSRAM; activations and norm weights in SRAM
- Per-token flash access: ~6 rows, ~450 bytes read from flash per generated token
- Performance: 9.88 tokens/second end-to-end (94.9 ms/token compute) for TinyStories; Barista model runs at roughly double that rate
- License: MIT; repository includes training, ablation, and quantization code in research/tinystories/ and full measurements in RESULTS.md
- Deployment: Two-step process — scripts/fetch_model.sh downloads and verifies model assets via SHA-256 and byte-size pins; scripts/deploy.sh compiles and flashes firmware from local artifacts without network access
The core innovation is the application of Per-Layer Embeddings (PLE), a technique introduced in Google's Gemma 3n models, to a microcontroller memory hierarchy. In typical LLM inference, embedding tables dominate parameter count but are read-only lookup structures. By placing these tables in slow but capacious flash and fetching only the few rows needed per token, the system avoids the throughput collapse that usually accompanies weight offloading to storage. The ESP32-S3's memory tiers — SRAM (fast, tiny), PSRAM (medium), flash (huge, slow) — each serve the data accessed at their natural frequency, turning the chip's constraints into an architectural feature.
Two models ship with the repository. The TinyStories model generates short synthetic stories and serves as the primary benchmark. The Barista model is a purpose-built variant that answers questions exclusively about espresso preparation, demonstrating that the same memory architecture can support different specialized heads. Both models occupy the board one at a time; deploying a new model replaces the previous one. The fetch script cross-checks the release's own metadata.json against pinned hashes and sizes, ensuring that a failed or corrupted download leaves the existing installation untouched.
Why It Matters
Running a language model on a sub-$10 microcontroller changes the cost floor for local AI inference. Until now, the cheapest practical platforms for on-device LLMs were single-board computers like the Raspberry Pi or smartphones, which cost an order of magnitude more and consume significantly more power. The ESP32-S3 draws milliwatts at idle and can operate on battery for extended periods, opening possibilities for always-on, privacy-preserving language interfaces in sensor nodes, wearables, and disconnected edge devices. The MIT license and complete source availability — including training scripts and quantization pipelines — mean the technique can be adapted to other small-model tasks without vendor lock-in.
The memory-tiering approach also has implications for larger models on constrained hardware. Google's Gemma 4-E2B-it, a 5.1-billion-parameter vision-language model released in April 2026, uses the same 4-bit quantization and PLE offloading to fit into just over 1 GB of memory, with further quantization pushing it toward 500 MB. While that footprint remains far above what an ESP32 can provide, the principle — keeping read-heavy embedding tables in slow storage while compute-intensive weights stay in fast memory — scales across device classes. For enterprises deploying AI at the edge, this reduces the bill-of-materials cost and power budget for a wide range of inference workloads that do not require frontier-model generality.
Geopolitically, the demonstration underscores that AI capability is diffusing to commodity hardware well below the threshold of export-controlled compute. An ESP32-S3 is a widely available, Chinese-manufactured microcontroller (Espressif Systems, Shanghai) with no licensing restrictions. The techniques used — quantization, embedding offloading, memory-hierarchy-aware scheduling — are published in open literature and implemented in open source. This suggests that the ability to run useful, if narrow, language models locally is becoming ubiquitous regardless of GPU access policies.
Our Take
The slvDev project is a genuine engineering milestone: it proves that a transformer-based language model can generate coherent text at human-readable speed on a chip designed for sensor fusion and Wi-Fi provisioning, not matrix multiplication. The 9.88 tokens/second figure is not a theoretical peak but a measured end-to-end rate including flash reads, display updates, and memory management. That it exceeds average human reading speed makes the interaction feel responsive rather than demonstrative.
However, the limitations are structural, not incidental. The TinyStories model's reasoning core is only about 4 million parameters — the rest is embedding tables that the PLE technique offloads. The memory trick does not make the model smarter; it only makes a larger model fit. Consequently, the system cannot generalize beyond its training distribution. It will not become a chatbot, a coding assistant, or a fact retriever without a fundamentally larger reasoning core, which in turn demands more fast memory than any current microcontroller provides. The Barista variant illustrates the trade-off: by narrowing the output domain to espresso Q&A, it achieves higher token rates but remains a party trick rather than a product.
For practitioners, the immediate value is the reference implementation. The repository's src/ directory contains a reusable inference engine that respects the ESP32-S3's memory hierarchy, and RESULTS.md documents ablations that quantify the impact of each design choice (quantization level, flash row count, PSRAM vs. SRAM placement). Teams building edge AI on similar SoCs — whether Espressif, Nordic, or TI — can adapt this architecture rather than starting from scratch. The two-step fetch/deploy model with cryptographic verification is also a sound pattern for over-the-air updates in production fleets.
We expect the next wave of microcontroller LLMs to focus on domain-specific distillation: taking a larger teacher model and compressing its knowledge into a tiny student that fits in SRAM+PSRAM without flash offloading, thereby recovering both speed and generality within a narrow task. The PLE technique will remain relevant for the embedding-heavy layers of those students. But for now, the slvDev demo stands as a proof that the memory wall is permeable — if you are willing to accept a model that only tells stories.
FAQ
What hardware exactly does this run on, and how much does it cost?
The model runs on an ESP32-S3 development board with 512 KB of SRAM, 8 MB of PSRAM, and 16 MB of flash storage. Compatible boards are widely available from distributors for under $10; the ESP32-S3 chip itself is manufactured by Espressif Systems in Shanghai.
How does the model fit in so little RAM?
Two techniques combine to make it fit. First, 4-bit quantization reduces the 28.9M parameters from ~60 MB (at 16-bit) to 14.9 MB. Second, Per-Layer Embeddings borrowed from Google's Gemma 3n move the 25M-parameter embedding table (~12 MB at 4-bit) into flash storage. Only ~6 rows (~450 bytes) are read from flash per generated token, leaving about 2 MB of active weights in PSRAM and SRAM.
Can this model answer questions or write code?
No. The TinyStories model was trained on a synthetic dataset of simple stories and can only generate short, coherent narratives. It lacks instruction-following, factual knowledge, and coding ability. The separate Barista model answers questions only about espresso preparation. The developer states the limitation comes from the small reasoning core (~4M parameters) and that the memory offloading technique does not change the model's capabilities.
Is the performance of 9.88 tokens/second practical for real applications?
For the narrow task of generating readable stories on a battery-powered device with no network, yes — it exceeds typical human reading speed. For any interactive or general-purpose use, no. The model cannot hold a conversation, follow prompts, or retrieve information. The architecture is a proof of concept for memory-tiered inference, not a deployable assistant.
Where can I find the code and try it myself?
The full project is on GitHub at github.com/slvDev/esp32-ai under an MIT license. It includes two deployment scripts: fetch_model.sh (downloads and verifies model artifacts) and deploy.sh (compiles and flashes firmware). Firmware READMEs for each model live in firmware/esp32_tinystories/ and firmware/esp32_barista/. Training and quantization code is in research/tinystories/.
