Hypernetwork injection scales better than LoRA for LLM knowledge

Injecting factual knowledge into large language models (LLMs) reliably at scale is an open challenge. A team of researchers (Nischay Dhankhar, Dos Baha, Abulhair Saparov) proposes using hypernetworks for train-time knowledge injection: given a large corpus of facts, a hypernetwork generates a fixed LoRA adapter that, when inserted into the target model, enables answering questions about those facts. They decouple the hypernetwork's injection capacity from the target model's general capability, enabling the first rigorous study of scaling laws for hypernetwork architectures.

To support this, they constructed MegaWikiQA, a large-scale dataset with tens of millions of multi-hop question-answer examples across 39 domains from Wikidata5M. Their results reveal power-law scaling along all architecture axes (hypernetwork depth, width, target network size) for loss and reasoning accuracy. Crucially, hypernetwork-based injection exhibits reliable out-of-distribution (OOD) generalization at increasing scales, with steeper scaling exponents than LoRA fine-tuning or full fine-tuning. These findings position hypernetworks as a principled, scalable substrate for train-time adaptation, offering a promising alternative to existing methods.