Integrated Gradients reveal pathogen vs. protective signals in BiomeGPT microbiome models

BiomeGPT-style transformers tokenize microbiome samples by fusing a fixed species identity (S) with a sample-specific abundance measurement (A), forming tokens T = S + A. Prior interpretability work ranked important tokens using attention weights from the special [CLS] token, but those weights are nonnegative—so they can't separate microbes that push a sample toward disease from those that push toward health. They also act after token fusion, obscuring how species identity and abundance independently affect the classifier's output.

Oren Nelson's new paper addresses these gaps with Integrated Gradients (IG), a signed, fusion-aware attribution method. He introduces a source-derived baseline T' = S + A₀ that fixes the species identity as a biological coordinate while isolating abundance variation. Applied to a disease-versus-health decision margin, this yields clear polarity: pathogenic and protective microbial signals are explicitly separated. The paper also proposes second-order Integrated Hessians to uncover community interaction rules—how perturbing one microbe alters the model's sensitivity to another, and which species tip ambiguous cases toward disease or health at a given abundance. This is the first principled approach to explainability for feature-tokenized transformers, generalizing beyond BiomeGPT to any smooth, differentiable model of this class. Code and a 15-page, 6-figure paper are available on arXiv.