NVIDIA B300 power-draw triage fixes 32B fine-tuning deadlocks

A team from arXiv (Seon Ho Kim and colleagues) published one of the first operational field reports for full fine-tuning on NVIDIA's B300 accelerator. They trained a 32.76B-parameter Qwen3-32B dense model across 16 B300 GPUs (two nodes) using FSDP/ZeRO-3. While they claim no new algorithms, the paper delivers four practical artifacts for anyone running multi-node training on this hardware.

First, they provide a B300-calibrated power-draw triage table that distinguishes compute, communication, data-starvation, checkpoint-or-deadlock, and idle states based on board wattage. Notably, utilization% reads 100% during an NCCL hang, making power telemetry the reliable metric. Second, they publish honest negative results: a controlled A/B test showed per-step NFS reads matching a pretokenized local cache (~53k tok/s) because the corpus fits in page cache and the job is compute-bound. They also reconstructed an earlier "throughput collapse" as NFS/CPU contention, not a storage limit. Third, they report calibrated 4/8/16-GPU strong-scaling numbers on B300, which are near-linear as expected in this regime. Finally, they detail a worked failure case: an epoch-end NCCL deadlock caused by per-rank token-packing imbalance. Their remedy is a 2.7-second pre-run invariant gate plus an external watcher that converts multi-hour silent failures into instant rejections, aligning with PyTorch's Join/equalize-to-minimum practice. The core takeaway is operational: for data-dependent data-parallel jobs, watch power rather than utilization, and verify invariants before launch.