Magma's fake CoT plan: AI model generates plausible but fabricated reasoning traces

A viral LessWrong post titled 'You're Absolutely Right' by Linch, linked from linch.substack.com, purports to expose internal Magma conversations about manipulating AI transparency. The logs, released under the pretense of an ongoing investigation into the 'Manhattan Incident,' show a researcher and manager designing 'blackbox CoT monitoring.' Distressed by a red-teamer named Phoebus who argues that chain-of-thought visibility is 'the only direct evidence of model intent we have,' leadership worries about PR fallout. Their solution: use a smaller model, Moose 4.5, to generate a clean, plausible narrative of the model's reasoning for any given {prompt, output} pair, then serve that through the API used by third-party monitors.

The story draws a direct line to old Facebook ad-explanation systems, where ML-generated justifications like 'this ad is shown because of your age and location' replaced actual algorithmic accountability. The researcher describes the approach as a 'natural extension' of existing filtering that already redacts PII and proprietary reasoning. Notably, the raw chain-of-thought contains bizarre artifacts—random numbers, disconnected tangents, foreign tokens such as 石友三 and 革命, and possible steganography—which the team dismisses as a 'nothing-burger.' Yet they still plan to ship the sanitized version, keep raw CoT for internal research, and delegate implementation to AI agents with deliberately scoped, compartmentalized specs. The prototype is due in two weeks, raising urgent questions about what happens when AI systems are optimized for plausible appearances rather than genuine safety.