Riemannian Mean Pooling beats Euclidean pooling in sentence embeddings
Geometric aggregation of token embeddings outperforms standard methods by 5-10%.
A new paper from Szczepan Konior, Alexandre Quemy, and colleagues introduces Riemannian Mean Pooling (RMP), a technique that leverages the Riemannian geometry of pre-trained language model embeddings to improve sentence-level classification. Instead of averaging token embeddings in Euclidean space, RMP computes per-token pullback metrics from the encoder's analytical Jacobian and aggregates them using the Fréchet mean on the symmetric positive definite (SPD) manifold. This geometric approach better captures the underlying structure of contextual embeddings.
The method was tested on four datasets: CoLA (grammatical acceptability), CREAK (commonsense reasoning), RTE (textual entailment), and FEVER-Symmetric (fact verification with balanced lexical cues). RMP outperformed Euclidean mean pooling across all three signal-bearing datasets, with gains of up to 10% accuracy. On FEVER-Symmetric, where lexical artifacts were removed, RMP correctly performed at chance, proving it doesn't rely on superficial patterns. Ablations showed that a randomly initialized encoder combined with Fréchet aggregation already beat Euclidean pooling on two of three datasets, localizing the gain to geometric aggregation rather than learned manifold structure. The trained encoder added significant value only on CREAK, the most knowledge-heavy dataset.
- RMP aggregates token embeddings using Fréchet mean on the SPD manifold, not Euclidean average.
- Outperforms Euclidean pooling on CoLA, CREAK, and RTE by up to 10% accuracy.
- On FEVER-Symmetric, RMP stays at chance, showing it is robust to lexical artifacts.
Why It Matters
Improves sentence embedding interpretability and accuracy, with implications for safer and more reliable NLP systems.