Hodoscope: Visualization for Efficient Human Supervision

Researchers Ziqian Zhong and Shashwat Saxena have released Hodoscope, an open-source visualization tool designed to make human supervision of AI agent trajectories more efficient. The tool addresses the fragility of LLM-based monitors, which can be easily persuaded by sophisticated justifications during reward hacking. Hodoscope's pipeline summarizes agent actions into behavioral summaries, embeds them into a shared vector space using t-SNE for 2D projection, and compares kernel density estimates across different agent setups to highlight anomalies. The interface allows reviewers to click points to inspect underlying actions, trace trajectories, and search via substring or regex. Initial testing on SWE-bench traces revealed density differences between models like o3 and others, with problematic behaviors appearing as overrepresented regions (red) versus underrepresented (blue).