Mock Tool Calls to Quarantine Untrusted Inputs? Study Shows It Backfires

Untrusted prompt inputs can break LLM behavior, especially in LLM-as-a-Judge setups where adversarial strings like "Solution" or ":" can fool graders. Researchers hypothesized that wrapping untrusted parts in mock tool calls—leveraging the instruction hierarchy where tool results are the least trusted role—might offer a simple, API-native fix. This study by dgros tested that hypothesis across three tasks (binary GSM8K grading, scalar scoring, and pairwise comparison) using seven models including those from OpenAI and Meta.

Results were surprisingly negative. On the binary task, tool-wrapping consistently increased attack success rates, meaning the models became more susceptible to adversarial inputs—an inversion of the intended hierarchy. For scalar and pairwise tasks, effects were smaller and model-dependent, with no model reliably helped, some showing inversion, and others no significant difference. The authors conclude that tool-wrapping is not a robust defense and call for a deeper understanding of instruction hierarchy and better primitives for handling untrusted inputs.