Deep learning framework maps leaf vein genes to environment in poplars

Leaf veins are incredibly diverse, but most gene–environment association studies have relied on a few low-dimensional summary traits, discarding most structural information. In a new arXiv paper, researchers from the University of Utah and collaborators present an integrated deep learning and statistical framework that treats the entire leaf vascular network as a whole-network image phenotype. The team fine-tunes the Edge Detection with Transformers (EDTER) model—trained on a custom annotated dataset derived from DiffusionEdge and the Berkeley Segmentation Database (BSDS500)—to extract complete vein architectures from RGB images, capturing both local and global contextual features. This allows the framework to represent the full complexity of leaf venation rather than reducing it to simple metrics.

To link these high-dimensional image features to genetic and environmental factors, the framework employs Semiparametric Sparse Canonical Correlation Analysis (SSCCA). This method handles variable selection and models associations between repeated measurements of bivariate image responses and high-dimensional predictors, while accommodating sparse, zero-inflated data via a truncated latent Gaussian copula model. Two simulation studies validated the framework's performance under increasing complexity. When applied to a real Populus (poplar) dataset, the approach identified three significant gene–geography interactions associated with leaf vascular architecture—findings that were previously obscured by coarse trait summaries. The authors position this work as a generally applicable methodological blueprint for analyzing high-dimensional complex image phenotypes in genomics and beyond.