Charon simulator predicts LLM performance with under 5.35% error
Researchers built a simulator that tunes LLM training and inference 3.74% error on clusters.
Deploying large language models at scale requires navigating an enormous design space of parallelism strategies, system optimizations, and hardware configurations. To help engineers and researchers prototype configurations without costly real-world trials, a team of researchers created Charon, a unified, modular, fine-grained simulator for LLM training and inference. Unlike fragmented tools, Charon models the entire stack—from data parallelism to pipeline parallelism, memory bandwidth, and interconnect—enabling accurate what-if analyses before any hardware is provisioned.
Charon's accuracy is remarkable: overall prediction error stays under 5.35%, and for large GPU clusters it dips below 3.74%. In a real-world inference deployment test, Charon discovered a configuration that improved system throughput over an existing engineering-tuned baseline, proving its tangible value. Accepted at MLSys 2026, Charon gives AI teams a fast, reliable way to simulate and optimize massive model deployments without burning GPU hours or cloud budgets.
- Charon simulates both training and inference for LLMs with end-to-end, fine-grained modeling.
- Overall prediction error <5.35%; training on large clusters error <3.74%.
- In a practical case, Charon beat an engineer-tuned baseline, improving throughput.
Why It Matters
Charon enables cost-effective optimization of LLM deployments without expensive trial-and-error on real hardware.