Balancing Latency and Accuracy of Code Completion via Local-Cloud Model Cascading
New framework uses a 121M-parameter local model to slash cloud LLM usage by 46.3% while improving accuracy.
A research team has introduced MCCom (Model-Cascading-based code Completion), a novel framework designed to solve the persistent trade-off in AI-powered code completion. The system intelligently cascades a small, fast language model (SLM) running locally with a powerful, but slower, large language model (LLM) in the cloud. The key innovation is using real-time user actions—like hesitation or deletion—as a signal to decide when to call the expensive cloud LLM, preventing unnecessary and costly queries.
MCCom employs a two-stage speculative decoding strategy and an iterative retrieval mechanism to enhance the collaboration between the local and cloud models. The team also trained a specialized 121M-parameter lightweight model that achieves 73.8% of the performance of a much larger 7B-parameter model. Evaluated on the RepoEval benchmark and a new real-world dataset called StmtEval, the framework demonstrated significant efficiency gains, reducing overall inference latency by up to 47.9% and cutting cloud LLM usage by 46.3%, all while improving the LLM's exact match accuracy by 8.9%.
- Cuts inference latency by up to 47.9% by cascading a local SLM with a cloud LLM.
- Reduces costly cloud LLM usage by 46.3% by using user actions as a trigger signal.
- Improves the cloud LLM's exact match rate by 8.9% through enhanced model collaboration.
Why It Matters
This makes professional-grade, accurate AI code assistants faster and more cost-effective to deploy at scale for developers.