ShardTensor: Domain Parallelism for Scientific ML at Extreme Scales

Scientific Machine Learning (SciML) struggles with extreme-resolution data—existing methods either fail to scale or degrade model accuracy. ShardTensor, introduced by Corey Adams and colleagues from NVIDIA and academia, offers a novel domain parallelism paradigm that fundamentally decouples the spatial dimensionality of input data from hardware constraints. This allows data of arbitrary size to be processed with no batch size restrictions per device. The framework supports both strong and weak scaling: strong scaling reduces inference latency, while weak scaling increases the total data volume a model can ingest. Crucially, ShardTensor enables parallelization across multiple spatial dimensions simultaneously, removing a key barrier for SciML on terascale or petascale inputs.

In experiments, ShardTensor demonstrated linear strong scaling up to 64 GPUs and near-ideal weak scaling for climate modeling and fluid dynamics datasets. Unlike naive domain decomposition, ShardTensor preserves training fidelity by maintaining gradient flow across shard boundaries. This approach is particularly impactful for applications like weather prediction, fusion plasma simulation, and astrophysics, where models must process global-scale data without compromise. The paper, available on arXiv, includes 10 pages of methodology and results across 7 figures and 2 tables, showing the framework’s generality beyond specialized implementations. ShardTensor represents a practical step toward making extreme-resolution SciML accessible without specialized hardware or custom coding.