Ray 2.56.0 boosts LLM serving, GPU placement, and data stability
New GPU-domain placement groups and 2x faster LLM serving with decoupled token streaming.
Ray 2.56.0 brings major stability and performance upgrades across its core, serve, and data modules. Ray Data now supports running multiple datasets in a cluster via subcluster labels and resource partitioning, plus automatic batch size selection (`batch_size='auto'`) in CPU-based map batches to prevent out-of-memory errors. Iter_batches stability is tightened by reducing hidden buffering and shutting down the executor when consumers exit early, reducing object-store spilling. A new DataSourceV2 framework debuts with ParquetDatasourceV2 and chunked reading, and Iceberg now supports distributed upsert using a task-based merge algorithm to avoid driver bottlenecks.
Ray Serve LLM has been re-architected by decoupling request handling from the token streaming response path, yielding substantial performance gains for LLM serving. New routing policies include session-sticky routing via consistent hashing (ConsistentHashRouter) and CapacityQueueRouter for supply-constrained workloads. Ray Core adds GPU-domain-aware placement groups using label locality, allowing bundles to pack onto nodes sharing a `ray.io/gpu-domain` label rather than only at the single-node level. Additionally, initial Kubernetes in-place pod resizing support for Autoscaler v2 enables CPU and memory resizing on existing worker pods before scaling out new pods.
- Ray Core: GPU-domain-aware placement groups and K8s in-place pod resizing for Autoscaler v2.
- Ray Serve LLM: Decoupled request handling and token streaming, session-sticky routing via ConsistentHashRouter.
- Ray Data: Multi-dataset clusters, auto batch size, default logical memory, Iceberg distributed upsert, and DataSourceV2 framework.
Why It Matters
Ray 2.56.0 significantly improves GPU utilization, LLM serving efficiency, and data pipeline stability for distributed AI workloads.