Flyback's top LightGBM feature actually hurt predictions — here's why

Flyback, a pricing engine for secondary market watches, recently uncovered a classic gradient boosting trap that turned their #1 feature into a liability. The team engineered a variant-conditioned Bayesian target encoder to isolate within-reference pricing dynamics. LightGBM quantile regression immediately loved it: the encoder ranked first in feature importance at the q90 quantile, with gains several times higher than the next best feature, consistent across multiple seed runs. But when the team ran a strict 4-seed × 3-variant ablation on the hold-out set, the results inverted. Test MAPE regressed by +0.28 percentage points, and the between-variant delta was seven times the within-variant standard deviation — a clear signal of overfitting.

The root cause? The encoder was finding effective splits during training that completely failed to generalize because the signal it learned was driven by irreducible label variance. Factors like subtle condition nuances, seller behavior, and timing are inherently unobservable in the feature set, so the model memorized noise rather than true patterns. This divergence underscores a critical lesson for practitioners: high feature importance does not guarantee generalization, especially with target encoding in gradient boosting. Flyback shared the full architecture, ablation methodology, and mechanism behind the divergence in a detailed post, urging teams to validate top features with rigorous cross-validation and ablation studies before trusting them in production.