[R] KALAVAI: Predicting When Independent Specialist Fusion Works (gain = 0.82 × divergence − 2.72, R² = 0.856, tested 410M–6.9B)

Researchers at Murai Labs have introduced KALAVAI, a novel method for fusing independently trained specialist AI models into a unified system that outperforms any single contributor. The approach involves taking a base model checkpoint (tested on Pythia models from 410M to 6.9B parameters), distributing copies to different teams who fine-tune them on private datasets without communication, then combining them using a lightweight MoE router trained in just 500 steps. The fused model consistently outperformed individual specialists by 7-8% at smaller scales and 6.5% at 6.9B parameters, with one 20-contributor experiment showing a 16.71% improvement.

The breakthrough includes a predictive formula (gain = 0.82 × divergence − 2.72, R² = 0.856) that estimates fusion benefits before training begins. In cross-lingual tests, specialists trained on Tamil, Yoruba, Welsh, and Code—languages the base Pythia model didn't know—were fused, reducing Yoruba perplexity from 41.9 to 7.7 and Welsh from 102.7 to 22.1. The router autonomously discovered domain relationships, like routing medical and chemistry text 60/40 without explicit instruction. The method requires full fine-tuning rather than LoRA and scales inference costs linearly with specialist count, but enables privacy-preserving collaboration where data cannot be shared.

Currently targeting NeurIPS 2026, the researchers are seeking independent validation, particularly at scales above 6.9B parameters. The approach is especially promising for low-resource language applications and domains with sensitive data, as it allows multiple organizations to contribute expertise without exposing their datasets. All code and scripts are available on GitHub for reproduction.