REDI framework automates data readiness for scientific AI across domains

A team of 11 researchers from Oak Ridge National Laboratory and other institutions released REDI (Readiness for AI), an open-source framework designed to automate the tedious process of preparing large-scale scientific datasets for AI training. Currently, no existing tool fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment—gaps REDI fills with a five-stage pipeline: ingest, preprocess, transform, structure, and output. Each stage includes instrumentation for reproducibility, and the framework can be deployed as a skill callable by AI agents. A companion tool, SetGo, automates FAIR compliance and catalog publication, ensuring datasets are findable, accessible, interoperable, and reusable.

REDI was evaluated across four scientific domains—climate modeling, proteomics, materials science, and nuclear fusion—successfully transforming all datasets from raw to AI-ready, with outputs validated against domain-expert references. Preliminary results show near-ideal parallel scaling to 100 nodes on the Frontier supercomputer for the climate case. Provenance-instrumented profiling revealed that file I/O is the dominant pipeline cost, and format selection is a first-order optimization lever. By automating data readiness, REDI transforms data preparation bottlenecks into reproducible, reusable community assets for scientific AI.