ParBench: New benchmark reveals LLMs struggle with parallel code translation

Researchers from Intel and academic institutions have released ParBench, a benchmark framework designed to rigorously evaluate how well large language models (LLMs) can translate parallel code between different programming APIs such as CUDA, OpenMP, OpenCL, and OpenMP target offload. The framework addresses a critical gap: existing evaluations often test on surface-level code similarity rather than verifying that low-level parallel semantics—like thread indexing, synchronization, memory management, and host-device coordination—are preserved. ParBench fixes the build, run, and verification infrastructure using declarative specifications, asking models to translate only the computational kernels while keeping the boilerplate consistent. To prevent models from succeeding through memorization, the benchmark includes AST-driven, intended behavior-preserving source augmentations that perturb the code while maintaining its original semantics.

Evaluations on state-of-the-art open and proprietary LLMs reveal persistent barriers to reliable parallel code translation. Key findings include direction asymmetry (e.g., translating from CUDA to OpenMP is harder than the reverse), multi-file coordination failures where models cannot maintain consistency across separate kernel files, incomplete API adaptation (like missing the target region syntax in OpenMP offload), and uneven robustness to source-level perturbations. The 57-page report (with 22 figures and 11 tables) provides a detailed analysis of these failure modes. ParBench's code is publicly available, offering a standardized tool for the community to benchmark progress. The work underscores that while LLMs are increasingly proposed for automated accelerator migration, they still lack the reliability needed for production use in high-performance computing.