AlphaZero Model Struggles with Othello Training Performance
Despite adjustments, AlphaZero's Othello model sees low win rates against benchmarks.
In an attempt to train an AlphaZero model for Othello on a 6x6 board, the user began with a c_puct value of 4.0, later reducing it to 3.5 after observing the model's performance. To encourage exploration, Dirichlet noise was applied with alpha set to 0.15. Despite these adjustments, the model's win rate against classical Monte Carlo Tree Search (MCTS) and a greedy agent remained dismally low, below 10%. The validation data revealed that value predictions were stagnant, prompting concerns about the model's learning capacity.
The user also analyzed the normalized entropy of predictions and the Kullback-Leibler divergence between successive models. While later models showed improvement over earlier versions, the lack of significant progress against benchmarks remained troubling. The Kullback-Leibler divergence stabilized quickly, indicating potential issues with how the model approaches learning and decision-making. The user is left questioning whether hyperparameter choices were optimal and how the statistical properties of the training data might explain the agents' poor performance. Insights from this analysis could drive future adjustments in training methodologies for AlphaZero models.
- Initial c_puct set at 4.0, reduced to 3.5 to enhance exploration.
- Model achieved under 10% win rate against benchmarks, indicating learning issues.
- Normalized entropies and KL-divergence metrics suggest potential flaws in training strategy.
Why It Matters
Understanding model training challenges can improve AI performance and decision-making in games.