PoolBench reveals flaws in LLM pooling strategies
New benchmark PoolBench shows 'last token' pooling lags behind in LLM concept tasks
Researcher Ayushi Agarwal has introduced PoolBench, a benchmark designed to rigorously evaluate pooling strategies in decoder-only large language models (LLMs) like Meta's Llama-3.1-8B, Google's Gemma-2-9B, and Mistral AI's Mistral-7B. The benchmark addresses a critical gap in LLM research by isolating pooling—how hidden states are collapsed into passage-level vectors—as the sole experimental variable under a fixed evaluation protocol.
PoolBench evaluates 19 pooling strategies across 17 concepts and 37,693 real-text passages, using linear separability (AUROC) as the primary metric. The results decisively favor W4_hierarchical pooling, which achieves a cross-model mean AUROC of 0.7799, outperforming the widely used P1_last_token baseline (0.7640) with statistical significance (p = 2.0e-36). The benchmark also reveals that pooling choice has a smaller impact (delta AUROC 0.016) than concept construction methods (delta AUROC 0.15), challenging prior assumptions. Agarwal releases the corpus, pre-extracted activations, and evaluation tools to standardize future research.
- PoolBench evaluates 19 pooling strategies across Llama-3.1-8B, Gemma-2-9B, and Mistral-7B using 37,693 real-text passages.
- W4_hierarchical pooling achieves 0.7799 AUROC vs. 0.7640 for P1_last_token, with statistical significance (p = 2.0e-36).
- Benchmark releases corpus, activations, and code to standardize pooling research in LLMs.
Why It Matters
Fixes a critical bottleneck in LLM concept representation, enabling more efficient and accurate AI systems.