Inkling-Small by Thinking Machines hits 1M context with just 12B active
A 276B-parameter MoE that runs on a single GPU thanks to 12B active and 1M context.
Thinking Machines has unveiled Inkling-Small, a massive Mixture-of-Experts (MoE) language model designed to deliver frontier-level performance without requiring data-center-scale hardware. The model packs 276B total parameters, but only 12B are active during any single forward pass—a sparse architecture that drastically reduces compute and memory requirements. Even more striking is its 1M-token context window, which allows the model to process entire codebases, lengthy documents, or long-running agent conversations in one go. For context, that's roughly 750,000 words, placing it in the same league as GPT-4.1 and Claude 3.5 in terms of context capacity, yet far more accessible in terms of inference cost.
Early adopters are already pushing the model's practicality. The official release includes an NVFP4 quantization on Hugging Face, which leverages NVIDIA's FP4 format to shrink the model footprint for RTX GPUs. In parallel, Unsloth published GGUF quants that enable llama.cpp compatibility. One developer reports successful local execution using a developmental llama.cpp branch with CUDA and CPU offloading—meaning the model can split its layers between a GPU and system RAM. This is a major milestone: a 1M-context MoE with 276B total parameters running on a workstation or high-end laptop, not a server cluster. For researchers and enterprises, it opens up private, cost-effective inference at a scale previously reserved for API-only models.
- 276B total parameters with 12B active (MoE) — runs on a single consumer GPU
- 1M-token context window, enabling whole-book or entire-codebase processing
- Unsloth GGUF quant works on llama.cpp with CUDA + CPU offloading for local use
Why It Matters
Professionals can now run a 1M-context MoE model locally, preserving data privacy while avoiding API costs.