[R] How are you managing long-running preprocessing jobs at scale? Curious what's actually working

A Reddit discussion is resonating with machine learning practitioners, highlighting a critical bottleneck in the AI development pipeline. Small ML teams, often focused on model architecture and training, are hitting a scalability wall when preprocessing large datasets ranging from 50 to 100GB. These jobs, running for hours on single machines, become major productivity sinks when they fail midway, forcing costly reruns. The post explicitly mentions evaluating workflow orchestration platforms like Prefect and Temporal, but finds they demand a dedicated DevOps skillset to implement and maintain—a resource many lean teams lack.

The core of the discussion revolves around practical solutions. Practitioners are asking peers if they've successfully distributed these compute-heavy jobs across multiple workers using frameworks like Apache Spark or Dask, and whether the performance gain justifies the steep setup and operational complexity. A key question is whether building a custom, internal solution is a worthwhile investment compared to wrestling with off-the-shelf enterprise tools. The conversation underscores a growing divide in the AI ecosystem: the need for robust, scalable data infrastructure versus the reality of teams whose primary expertise lies in algorithms, not distributed systems engineering.