Thinking Machines' Inkling: open-weight 70B model matches GPT-4o on key benchmarks
Inkling achieves 95% of GPT-4o performance at 10% inference cost, runs on consumer hardware.
Thinking Machines today unveiled Inkling, a fully open-weight 70B parameter language model released under Apache 2.0. Trained on 2 trillion tokens of high-quality web data and synthetic code, Inkling achieves 86.5% on MMLU and 82% on HumanEval, matching OpenAI’s GPT-4o within 5% on most reasoning tasks. Remarkably, it uses 40% fewer FLOPs during inference than comparable models like Llama 3 70B, thanks to a novel mixture-of-experts (MoE) architecture with 16 experts. The model runs on a single NVIDIA A100 80GB GPU at 30 tokens per second, or on consumer hardware with 4-bit quantization.
For enterprises, Inkling eliminates API dependency: sensitive data stays on-prem, latency drops below 100ms, and cost per million tokens is $0.15 (versus $2.50 for GPT-4o). Early adopters include legal tech firm LexAI, which used finetuned Inkling for contract review, and healthcare startup Medivance for clinical note summarization. The release includes a LoRA adapter, a chat-tuned variant, and a Ragged (retrieval-augmented generation) pipeline template. This move signals a shift toward democratizing frontier models, pressuring closed-source providers to justify their premiums.
- 70B MoE model outperforms Llama 3 70B with 86.5% MMLU, 82% HumanEval
- 40% less inference compute than comparable models; runs on single A100 GPU
- Apache 2.0 license, includes LoRA adapters, chat variant, and RAG starter kit
Why It Matters
Open-weight AI competitive with GPT-4o cuts costs 94%, enabling private, affordable enterprise deployment.