FollowTable: A Benchmark for Instruction-Following Table Retrieval
Existing retrieval models can't handle fine-grained constraints on tables, per SIGIR 2026 study.
A team of researchers (Rihui Jin, Yuchen Lu, Ting Zhang, et al.) has introduced FollowTable, a new benchmark designed to test how well retrieval models follow fine-grained instructions when searching over tabular data. Traditional table retrieval (TR) relies on topical semantic similarity, but the rise of LLM-based agentic systems demands instruction-driven retrieval where relevance depends on explicit content and schema constraints.
FollowTable formalizes this as Instruction-Following Table Retrieval (IFTR), identifying two core challenges: sensitivity to content scope (inclusion/exclusion constraints) and schema-grounded awareness (column semantics, representation granularity). Using a taxonomy-driven annotation pipeline, the team built the benchmark and proposed the Instruction Responsiveness Score to measure how retrieval rankings adapt to instructions relative to a topic-only baseline. Their results reveal that existing models are biased toward surface-level semantic cues and struggle with schema constraints, opening a clear gap for future improvements. The paper was accepted at SIGIR 2026.
- First large-scale benchmark (FollowTable) for instruction-following table retrieval, accepted at SIGIR 2026.
- Introduces a new metric: Instruction Responsiveness Score – measures ranking adaptation to user instructions vs. topic-only baselines.
- Existing models show systematic bias toward surface-level semantic cues and fail to handle schema-grounded constraints like column semantics and inclusion/exclusion rules.
Why It Matters
As LLM agents increasingly query structured data, this benchmark highlights a critical blind spot in current retrieval systems.