GPT-4o-mini features degrade GNN accuracy by 17% on PubMed benchmark
Simple concatenation of LLM embeddings can hurt graph models significantly.
A new study reveals that simply appending LLM-generated node features to Graph Neural Networks (GNNs) can backfire on standard homophilous graph benchmarks. Researchers from the paper (arXiv:2606.17579) show that concatenating SBERT-encoded GPT-4o-mini TAPE features with bag-of-words inputs reduces PubMed test accuracy by a dramatic 17.0±0.3 percentage points, and Cora by 4.3±0.6 points. The effect attenuates with stronger GNN backbones (GCN, GCNII, GAT) and reverses on medium-homophily datasets (WikiCS +4.4 pp, ogbn-arxiv +11.7 pp).
The authors introduce a simple measure called Delta_sig (LLM-alone discriminability) that predicts concatenation cost better than homophily (r²=0.38 vs 0.06, N=9). A dimension-controlled ablation shows the drop is not due to dimensionality or weight decay: same-source PCA only costs -2.3 pp, while Gaussian noise costs -37.3 pp. Nine configurations on PubMed fit a power law |Δ_concat| ∝ (√(d_l/n))^1.31 with r²=0.97, pinpointing exactly where the headline -17 pp occurs. The paper provides practical guidance for researchers using LLM features in graph learning.
- Concatenating GPT-4o-mini features to MLP+GNN on PubMed causes a -17.0±0.3 pp accuracy drop.
- Proposed Delta_sig measure predicts concatenation harm better than homophily (r²=0.38 vs 0.06).
- Power law (r²=0.97) describes the degradation; dimensionality and weight decay are ruled out as causes.
Why It Matters
Caution for graph ML practitioners: LLM feature concatenation isn't always beneficial and requires careful validation.