Scheduling Coflows in Multi-Core OCS Networks with Performance Guarantee
New scheduling method tackles the cross-core bottleneck in modern optical data centers, promising faster AI training.
A team of researchers has published a paper tackling a critical bottleneck in modern data centers: efficiently scheduling communication patterns, or 'coflows,' across multi-core Optical Circuit Switching (OCS) networks. As AI and big data workloads demand massive bandwidth, data centers are moving beyond single-core OCS to use multiple independent optical cores concurrently. However, existing coflow scheduling research has lagged, primarily focusing on single-core or electrical packet-switched networks. This creates a significant gap, as scheduling in a multi-core OCS fabric introduces unique challenges from cross-core traffic coupling and the physical constraints of optical circuit reconfiguration.
To solve this, the researchers propose a novel approximation algorithm that jointly optimizes two problems: assigning traffic flows across heterogeneous cores and scheduling the optical circuits within each core under 'port exclusivity' and 'reconfiguration delay' constraints. The goal is to minimize the total weighted Coflow Completion Time (CCT), a key metric for job performance in distributed systems like AI training clusters. The algorithm provides a provable worst-case performance guarantee, a crucial feature for reliable system design, and its framework is also directly applicable to multi-core electrical packet switching (EPS) scenarios.
The paper's significance is validated through trace-driven simulations using real-world Facebook workload data. The results demonstrate that the new algorithm effectively reduces both the average (weighted) and the worst-case (tail) coflow completion times compared to prior approaches. This translates to faster job completion for data-intensive applications, from large-scale model training to big data analytics, directly impacting the throughput and cost-efficiency of cloud and AI infrastructure. By providing a formal, high-performance solution for this emerging hardware paradigm, the research paves the way for next-generation data center networks to fully harness the potential of multi-core optical switching.
- Solves the previously unaddressed problem of coflow scheduling in multi-core Optical Circuit Switching (OCS) data center networks, moving beyond single-core research.
- Proposes an approximation algorithm with a provable performance guarantee that jointly handles cross-core flow assignment and per-core circuit scheduling to minimize Coflow Completion Time (CCT).
- Validated with real Facebook traces, showing effective reductions in both average (weighted) and worst-case (tail) completion times for distributed computing jobs.
Why It Matters
Enables faster AI training and data processing by optimizing communication in next-gen optical data centers, reducing job completion times and infrastructure costs.