Letting AI Use the Command Line Beats Fancy Tools, Study Says

Companies are racing to build "digital workers" — AI agents (AI that can take actions on its own) that answer email, move information between business apps, and run analyses. The open question: how should those agents be plugged into software? Researchers compared five approaches on two simulated office-work test suites, TheAgentCompany and APEX-Agents, using two leading AI models. One approach gave the AI a simple command line, the same text-based interface programmers use. Another gave it a tidy menu of pre-built buttons, each doing one specific job.

The command line won, and not by a little. It beat the button menu by roughly 22 to 25 points on one test suite and 5 to 7 points on the other — while burning 19% to 72% fewer tokens. Tokens are the chunks of text AI reads and writes; you pay per token, so fewer tokens means a cheaper bill. Stacking the button menu on top of the command line added no measurable improvement. More structure, it turns out, just got in the way.

Why? A command line lets the AI improvise, chain steps together, and handle surprises. A fixed catalog of approved actions boxes it in. The study also tested a middle path called programmatic tool calling, where the AI writes small programs but can only perform actions from an approved list. That used fewer tokens than calling tools one at a time, but generally lost to the command line on both quality and cost.

The practical takeaway has two sides. Companies can get better, cheaper AI assistants by giving them a command line instead of an elaborate toolkit. But a command line means unrestricted execution — the AI can run almost anything, which is a real security and compliance risk. The authors suggest the restricted approach when policy demands an approved action list. One caveat: this is a single study, on two test suites, with two specific models.