OSDF benchmarks XRootD and Pelican to develop data transfer best practices

The Open Science Data Federation (OSDF) project aims to build a global scientific data distribution network based on the Pelican platform, which itself relies on XRootD and Pelican technologies. To ensure optimal performance, the team urgently needed to understand XRootD's limits under different configurations—including transfer rate limits, buffer settings, and storage types. Researchers Fabio Andrijauskas, Igor Sfiligoi, and Frank Würthwein executed a comprehensive set of benchmarks using hosts from the National Research Platform (NRP). They tested a range of file sizes, parallel streams, and client distances from the server, covering both standalone clients (wget, curl, pelican) and native HTCondor file transfer mechanisms.

The benchmarks created a methodology to track how XRootD and the Pelican layer perform across various scenarios. The results will inform a set of best practices to be shared with both the XRootD and Pelican development teams. This work is critical for ensuring that OSDF can efficiently serve the growing data needs of scientific research, where data sharing across institutions is increasingly essential. The paper was submitted to arXiv on May 13, 2026, and is also published in EPJ Web of Conferences (DOI: 10.1051/epjconf/202533701256).