This Trick Makes AI Agents Reliable Enough for Finance and Law

A new study on AI agents shows that a specific form of structured planning can make their behavior dramatically more predictable—but not without trade-offs. Researchers tested "harness engineering," wrapping agents in constraints like forced tool selection, output validation, and bounded retry. An initial version only improved reproducibility in one of four model-task combinations, actually made it worse in two, and did nothing in the fourth. A trace-level diagnosis traced the remaining unpredictability to an unconstrained free-text planning step. Adding Structured Planning, which validates the plan against a fixed schema before any tool runs, eliminated the issue in most cases: three of four model-task combinations reached perfect reproducibility and determinism at 100 runs, and task success hit 100% in three of four. The gain isn't free: token costs dropped in every case, but latency split by model—one became faster, the other markedly slower. The findings point to harness engineering as a real reliability discipline for regulated domains such as finance and compliance, but one whose costs must be measured, not assumed.