Training LLMs for True Randomness: Solving Bias with Distributional Fidelity

Recent work exposes a critical flaw in LLMs: they fail at generating truly random outputs. When asked to pick a random weekday, Qwen3 selects Wednesday about 80% of the time. Gemma-3 names only four cities in 75% of its responses, and multiple-choice questions see models heavily favoring option C for correct answers. This bias stems from training objectives that optimize for maximum likelihood, collapsing probability mass onto narrow modes even when diversity is essential for synthetic data generation, creative tasks, and agentic behavior. Researchers have now proposed addressing this as a training-time problem rather than an inference-time hack.

To solve this, the team trained models on minimal prompts to generate numbers from 30 different distribution families (24 seen during training). They evaluated three dimensions: distributional fidelity (do samples match the target distribution?), transfer to natural-language settings (e.g., random city names, balanced MCQ positions), and retention of other capabilities. Results show the method significantly improves stochastic behavior without major performance degradation, offering a path toward more reliable and diverse LLM outputs for downstream applications like synthetic data pipelines and unbiased agents.