SANA framework reveals data analysis as key bottleneck for QA agents on data lakes

Exploratory question answering (EQA) over massive data lakes remains challenging for LLM agents, which must discover relevant sources, analyze retrieved data, and adapt their actions. End-to-end accuracy often conflates failures in search, planning, and data analysis. To address this, researchers from Columbia University and collaborators introduce SANA (Search Agent Navigation Ablation framework), presented on arXiv in June 2026. SANA transforms EQA tasks into runtime profiles containing gold source sequences, sanitized subquestions, and execution records. It then constructs idealized versions of each component—search, planning, and data analysis—allowing systematic ablation. The residual gap isolates failures in the agent's action policy (decisions about what to do next).

Evaluating lightweight and mid-sized agents on adapted versions of LakeQA (large data lake) and KramaBench (smaller scale) reveals clear patterns: data analysis is a consistent bottleneck across both benchmarks, while planning is less limiting. Search becomes a major limitation in LakeQA's large data-lake setting, but less so for KramaBench. SANA thus provides a reusable diagnostic tool that decomposes end-to-end accuracy into component-level insights, enabling researchers to target improvements in search, planning, data analysis, and agent design. The framework promises to accelerate progress in data-lake QA by highlighting where current agents truly fail.