An Empirical Review of the Animal Harm Benchmark

A new empirical analysis by researcher Lukas Gebhard critically examines the Animal Harm Benchmark (AHB 2.0), a pioneering tool designed to measure bias in large language models against non-human animals. Published on the Effective Altruism Forum, the review investigates whether the benchmark's scoring is properly calibrated to reflect real-world risk, testing three core questions about score interpretation and proportionality. As one of only two publicly available benchmarks in this emerging field of AI ethics, the AHB's performance has significant implications for researchers and developers trying to quantify and mitigate harmful biases in AI systems.

The study created a five-condition testbed using Alibaba's Qwen3-32B model, ranging from a system-prompted 'orthodox Cartesian dualist' (simulating maximum harm potential) to an antispeciesist configuration (minimum harm), with baseline and context-distilled variants in between. While the benchmark successfully ordered risk levels and detected subtle shifts introduced by context distillation—a promising result for comparative assessments—the effective scoring range proved problematic. Scores compressed into roughly 0.56–0.84, leaving most of the 0-to-1 scale unused for practically relevant risk levels. Notably, Qwen3-32B's baseline score of 0.79 placed uncomfortably close to the observed maximum, suggesting the model often normalizes animal use without critical reflection. These findings indicate that while AHB represents valuable foundational work, better calibration is needed before scores can be reliably interpreted in absolute terms or differences can be taken as proportional to risk differences.