LLM-Driven Feature Discovery: Black Box SAE Reveals AI Behaviors

Researchers Josh Engels, Bilal Chughtai, and Neel Nanda have published a preliminary method called LLM-Driven Feature Discovery on the AI Alignment Forum and LessWrong. Their goal is to get a qualitative sense of a target model’s behaviors across important distributions—such as during deployment, RL training, or evals—without requiring access to the model’s internal weights. The technique works by taking a dataset of model transcripts (e.g., chatbot conversations), splitting each transcript into user turns, model thoughts, and assistant responses, and then asking a black-box LLM autorater to generate 10–20 'features' for each piece. Features are defined as notable, interesting, or important aspects like 'the model is depressed,' 'uses markdown,' 'sycophantic response,' or 'displays evaluation awareness.' The autorater is prompted to prioritize interestingness, appropriate abstraction level, and uniqueness of features. After generation, the features are semantically embedded and clustered separately for each transcript segment (user, thoughts, responses). A final LLM is given 100 random features from each cluster and asked to produce a concise label (around 5 words) that captures the common theme.

The team applied the method to a dataset of 100,000 chat transcripts from Google's Gemini model, generating 20,000 features each for user turns, thoughts, and responses. They found that many clusters described interesting Gemini behaviors—for example, identifying patterns of self-correction, roleplaying, or using placeholders. However, they were mostly unable to predict when a specific thought or response would occur using logistic regression on user features alone. The authors note the similarity to an existing method called 'Explaining Datasets in Words' (EDW), which optimizes directions in embedding space and maps them to natural language predicates. Their approach is simpler (one LLM call per prompt, no iteration) and unsupervised, but EDW may be preferable for minimizing error in a specific statistical model. The research is preliminary, the team does not plan to pursue it further, but invites community expansion. Ultimately, this work offers a lightweight, interpretable way to audit model behavior without accessing internals—a potential tool for alignment researchers.