Dietz's nugget annotation tool redefines human-AI evaluation roles

Laura Dietz's new paper tackles a core flaw in LLM-as-a-judge evaluations: either human experts are unconsciously anchored (leading to rubber-stamping) or left swamped with cognitively demanding labeling. The proposed prototype annotation tool implements a smarter division of labor. Humans focus on identifying what information truly matters—small, self-contained 'nuggets' of relevance. LLMs then take over the high-volume task of matching these nuggets across multiple system outputs. This plays to each party's strengths while keeping genuine human oversight in the loop.

The design avoids common pitfalls like anchoring and overload. The paper details key decisions behind the human-AI workflow and how the accumulated nugget banks can be reused with automated judges. Instead of binary thumbs-up/down, evaluations become granular and transparent—each nugget becomes a traceable unit of quality. For teams running frequent LLM benchmarks or agent evaluations, this could mean faster, more reliable quality signals without expensive expert hours. The approach is especially relevant as agentic systems produce diverse, open-ended outputs that simpler evaluation methods can't handle.