Goodfire's Silico reproduces RLFR, challenging the 'Most Forbidden Technique' taboo

Goodfire has launched a private beta of Silico, an LLM training platform that reproduces RLFR (reinforcement learning from probes)—a technique that uses internal model probes as reward signals for reinforcement learning. This has rapidly drawn comparisons to the 'Most Forbidden Technique' concept from the classic LessWrong post, which warned against training directly on model internals due to risks of obfuscation and gaming of safety metrics. Goodfire's accompanying post, authored by Rauno Arike, argues that such blanket objections are overblown and that training on internals can be acceptable under specific conditions.

Arike's literature review cites The Obfuscation Atlas by Taufeeque et al., which provides nuanced guidance on when using model internals in training is warranted. The key red line, per alignment researcher Daniel Kokotajlo, is maintaining a held-out test set that is not trained on or correlated with training proxies. Arike emphasizes that probe-based signals (like RLFR) should not be categorically forbidden, but rather used with caution by ensuring validation measures remain uncorrelated with training objectives. Silico's release reignites the debate over alignment vs. safety, offering a practical tool while pushing the community to move beyond cached memes.