CodeAlchemy generates 500B+ tokens for code AI, 3B model beats 27B ones
A 3B parameter model outperforms models 10x its size on code tasks.
Deep Dive
CodeAlchemy transforms public code into 500B+ tokens of synthetic data across 15 languages using five strategies. CodeTrace executes 1.3M+ files. The framework introduces DevEval and TraceEval benchmarks. A 3B model achieved 83.5% on HumanEval, outperforming 27B Gemma-3 and 32B Granite-4.0. Frontier model Claude Sonnet 4.5 scored only 5.6% on TraceEval.
Key Points
- Generated 500B+ tokens of synthetic data plus 350B reasoning tokens using 5 rewriting strategies across 15 languages.
- Introduced DevEval and TraceEval benchmarks; Claude Sonnet 4.5 achieved only 5.6% exact match on TraceEval.
- A 3B model achieves 83.5% on HumanEval and 63.2% on MBPP, outperforming models 10x larger like 27B Gemma-3.
Why It Matters
Synthetic data at scale could democratize code AI, enabling smaller models to rival much larger ones.