Salesforce's AI Struggles to Write Its Own Code, New Test Reveals
Even the best AI can't build a chatbot from scratch on Salesforce's platform—what that means for you.
Salesforce's Agentforce lets companies build AI chatbots without writing traditional code—instead, they use a special language called AgentScript. But a new benchmark, AgentScript-Eval, found that even the most advanced AI models struggle with this language. In tests, the best AI completed only 25% of tasks correctly, and no AI could build a working agent from scratch. This matters because businesses hoping to quickly deploy AI agents may face delays and higher costs if they need human experts to fix AI mistakes.
The test included 100 real-world tasks like creating an agent from a description, fixing broken code, and improving existing agents. Surprisingly, a smaller, cheaper AI model performed as well as the biggest, most expensive ones. The key to better results wasn't a bigger model, but giving the AI a step-by-step process to check and fix its own work. This suggests that how you use the AI is more important than the size of the AI itself.
The study also found that the way instructions are given to the AI matters a lot. The same guidance scored 28% when given as plain text but only 14% when presented as an interactive plugin. Rewriting one instruction inside that plugin raised the score to 37%. For businesses, this means that investing in clear, well-structured guidance for AI can significantly improve results without upgrading to a more expensive model.
Overall, while AI can help with coding, it's not yet ready to replace human developers for specialized enterprise languages. Companies should plan for human oversight and may need to provide better training and tools for their AI systems. The findings highlight that the surrounding support system—not just the AI model—is crucial for success.
- Top AI models fail at writing code for Salesforce's Agentforce, with the best scoring only 25% on a new benchmark.
- A smaller, cheaper AI model matched the biggest models, showing that price and size don't guarantee better performance.
- How you guide the AI—like giving it a step-by-step process—matters more than the model itself, doubling accuracy.
Why It Matters
Businesses may need human experts to set up AI agents, slowing automation and increasing costs.