New study: LLMs lose cultural knowledge due to data funnel

A new paper from Ananya Sahu and five co-authors at arXiv challenges the dominant approach to cultural alignment in large language models. Titled "The Culture Funnel: You Can't Align What isn't in the Data," the study argues that current methods—which focus on inference-time interventions like prompt engineering or RLHF—assume models already contain sufficient cultural knowledge. Instead, the researchers demonstrate a systematic decline in cultural signals as data moves through the LLM pipeline: pretraining, fine-tuning, alignment, and reasoning. They applied a multidimensional tagging framework to trace explicit cultural markers across each stage, finding that post-training datasets are dominated by geographically concentrated, task-specialized content, effectively filtering out diverse cultural perspectives.

The team released a culturally tagged dataset of 5.6 million samples to support future work. Their experiments show that adding these tags improves performance on downstream cultural benchmarks, suggesting that the root cause of poor cultural alignment is data scarcity, not inference-time tweaks. While multilingual training does increase geographic coverage, it fails to ensure balanced representation within regions. The paper calls for a fundamental shift: instead of trying to "align" models after training, builders must curate training data that inherently reflects the world's cultural diversity. The findings have direct implications for developers deploying LLMs globally—without fixing the data funnel, no amount of post-hoc alignment can produce truly equitable AI.