Serverless MapReduce framework on Kubernetes scales to zero

Modern logistics systems generate continuous data streams from GPS, IoT sensors, and management platforms. Processing this data in real-time is critical for monitoring operations and optimizing decisions. A new paper introduces a serverless MapReduce framework designed for exactly this use case. The system runs on Kubernetes with Knative, leveraging Function-as-a-Service (FaaS) principles to achieve elastic scaling. Five loosely coupled services handle data ingestion, processing, and aggregation, with Apache Kafka acting as the communication backbone. Redis preserves workflow metadata, while AWS S3 provides persistent storage.

The framework's architecture is inspired by the classic MapReduce model but adapted for event-driven, serverless execution. Experimental evaluation demonstrates effective scaling as input data volume grows, including a scale-to-zero mode that eliminates idle compute costs. By decoupling components and using configurable auto-scaling based on workload and hardware, the system offers a practical solution for real-time logistics analytics. The authors highlight its potential for broader distributed computing scenarios where cost efficiency and low latency are paramount.