[R] Is autoresearch really better than classic hyperparameter tuning?

A new experimental comparison reveals that AutoResearch, an AI-driven approach to hyperparameter optimization, significantly outperforms the established Optuna framework. In tests conducted on the NanoChat model, AutoResearch demonstrated faster convergence, better cost-efficiency, and superior generalization of the final solutions. Crucially, researchers aligned the comparison by having Claude define Optuna's search space, ensuring both methods operated with similar prior knowledge. Despite AutoResearch having a 2× higher cost per optimization step, its superior sample efficiency made it the more economical choice across all tested budgets.

Beyond just tuning parameters, AutoResearch's key advantage lies in its ability to search directly within the code space. While it initially operates within Optuna's 16-parameter framework, it progressively explores more fundamental algorithmic changes as iterations increase. When the best solutions from each method were given additional training time, the performance gap widened, with AutoResearch's solutions showing statistically stronger results. This suggests the method isn't just finding better hyperparameters—it's discovering better algorithms.