New MORPH-1K benchmark tests moral sensitivity in 8 LLMs
LLMs distracted by irrelevant info still identify moral features accurately
Deep Dive
A new benchmark called MORPH-1K, a procedurally-generated 1,000-case set, evaluates moral sensitivity in LLMs using textual noise distractors that don't alter moral content. Testing on eight LLMs, the method avoids expensive human baselines or circular LLM judges. Results show noise changed the number of features listed, but semantic content remained stable above calibrated thresholds.
Key Points
- MORPH-1K uses 1,000 procedurally generated cases with 50 moral foundation-pole combinations across four social domains
- Textual distractors are validated to change only irrelevant details, not moral content
- 8 LLMs tested: perturbed inputs changed feature counts but semantic similarity stayed above threshold
Why It Matters
Scaling ethical AI evaluations without human baselines or circular LLM judges enables safer deployment.