LLaVA-Vicuna lesion study recovers aphasia parameters with 81.4% fidelity

Interpretability methods for large language models (LLMs) typically describe internal states but don't test whether those states causally produce observed behavior. In a new arXiv paper, researchers led by Yong Yang and Julius Fridriksson (University of South Carolina) address this by solving the inverse problem: given an aphasic picture-naming error profile, can they recover the exact lesion parameters that generated it? They lesioned LLaVA-Vicuna 13B across 4,840 configurations, varying layer index, modification percentage, and noise sigma, and classified errors into a seven-category clinical taxonomy (correct, semantic, unrelated, formal, mixed, neologism, no-response). A multi-task neural network was then trained to map these error profiles back to perturbation parameters. The results were partially successful: modification percentage and noise sigma were reliably recovered, while layer index was only localizable within a neighborhood.

Counterfactual validation proved more revealing: when fresh model instances were perturbed with the recovered parameters, they reproduced the target behavior in 81.4% of cases. This dissociation—poor layer recovery yet high behavioral fidelity—suggests functional redundancy across transformer layers, a property standard interpretability tools miss. As an out-of-distribution test, the inverse model was applied to real picture-naming error profiles from 278 stroke survivors. The recovered parameters were syndrome-discriminative, most strongly for perturbation intensity, indicating the method generalizes beyond synthetic data. The approach positions counterfactual validation as a robust framework for causal interpretability claims in LLMs, with potential applications in both AI alignment and clinical aphasia assessment.