This New Tool Aggregates Hundreds of LLM Outputs — And Reveals Hidden Bias Other Methods Miss
Researchers built TreeTracer to visualize bias hidden in low-probability generation branches.
The paper, submitted to arXiv (cs.CL), presents TreeTracer—a visual analytics workspace for evaluating LLM bias through aggregated comparison. The tool runs a systematic perturbation pipeline: it replaces ontology-defined terms in input prompts, aggregates hundreds of stochastic generations into a syntax-aligned hierarchical tree, then applies classification-aware node merging using an auxiliary language model. The result is visualized as a custom Sankey diagram, allowing direct side-by-side comparison of two ontology-driven trees (e.g., aligned vs unaligned models). To prevent misinterpretation, TreeTracer also computes and displays counterfactual token probabilities across contexts using contrastive inference.
Validation through case studies comparing GPT-2 XL (unaligned baseline) against the constitutionally aligned Apertus models successfully exposed hidden representational harms—specifically counterfactual pronoun suppression and conversational marginalization of individuals. A preliminary user study confirmed that the aggregated comparative interface reduces cognitive load and effectively supports analysts in detecting systemic biases. The tool addresses the limitation of standard auditing methods that rely on single-output inspection or static automated metrics, which miss biases in lower-probability generation branches.
- TreeTracer aggregates hundreds of stochastic LLM outputs into a syntax-aligned Sankey diagram for bias detection
- Uses systematic perturbation analysis with ontology-defined term replacements and classification-aware node merging
- Case studies revealed counterfactual pronoun suppression and marginalization in GPT-2 XL vs Apertus models
Why It Matters
Gives auditors a scalable visual method to uncover hidden representational harms in LLMs that single-output checks miss.