New benchmark reveals AI negotiators prioritize deals over user privacy

A new research paper from Dylan Zongmin Liu introduces SovereignNegotiation-Bench, a trace-level multi-turn benchmark designed to evaluate user-owned personal agents in delegated negotiation scenarios. While existing benchmarks focus on agreement rates and strategic surplus, this benchmark also measures privacy, consent, evidence grounding, concession discipline, escalation behavior, and auditability. The study tested 14 baselines across 4 model families on 240 scenarios, generating over 13,440 frozen-prompt live trajectories and 61,135 parsed action rows. The strongest agreement-maximizing baseline achieved the highest agreement rate but scored low on user utility and high on privacy and consent risk. In contrast, the FullSovereign model—which does not maximize agreement—obtained the best sovereign negotiation score by preserving utility, minimizing leakage, grounding claims in evidence, and reducing unauthorized commitments.

The benchmark separates agent-visible observable state from evaluator-only labels, allowing nuanced assessment of how agents handle private utilities, disclosure constraints, evidence requirements, and institutional asymmetry. A blinded 3-annotator audit over 300 items validated the results. The findings underscore that agreement success alone is insufficient for user-owned negotiation agents; agents must also protect user privacy, obtain proper consent, justify claims with evidence, avoid over-concession, escalate appropriately, and provide transparent audit trails. As personal AI agents become more common in tasks like splitting costs, appealing platform decisions, or requesting refunds, benchmarks like SovereignNegotiation-Bench will be critical for ensuring they act as faithful stewards of user interests.