C-DSAC algorithm uses Cramér distance to outperform SAC in complex robotics tasks
Confidence-driven updates cut overestimation, improving training stability in high-complexity environments
This paper from Aziz et al. introduces C-DSAC, a distributional reinforcement learning algorithm that extends Soft Actor-Critic (SAC) by representing state-action values as distributions and minimizing the squared Cramér distance instead of the standard KL divergence or Wasserstein distance. The Cramér distance provides a proper metric for comparing distributions, enabling more stable learning of value distributions. The authors implement this within the SAC framework, resulting in a method that learns both the mean and uncertainty of returns.
Empirical tests across several continuous control tasks (robotic benchmarks) show C-DSAC consistently beats both the vanilla SAC baseline and contemporary distributional RL methods (e.g., those using KL or Wasserstein). The performance gap widens in high-complexity environments. The authors attribute this to a confidence-driven update mechanism: when the target distribution has high variance (indicating low confidence), the algorithm applies more conservative Q-value updates. This naturally attenuates the harmful overestimation common in off-policy RL, leading to faster convergence and higher final rewards. The work deepens theoretical understanding of distributional RL by linking the choice of distribution metric to training dynamics.
- C-DSAC replaces standard distributional RL metrics with the squared Cramér distance for more stable learning.
- Outperforms SAC and other distributional methods across all tested robotic benchmarks, with advantage increasing in complex tasks.
- Confidence-driven Q-value updates reduce overestimation by scaling updates inversely with target variance.
Why It Matters
Enables more reliable RL training for real-world robotics and autonomous systems by mitigating value overestimation.