OpenAI's New AI Does Legal Research — But Gets Half the Answers Wrong
Lawyers get an AI research assistant — just don't trust it without checking.
OpenAI has released Astra for Law, a version of its newest AI model tuned specifically for legal work. Instead of searching the ordinary internet, it reads from a dedicated index of US legal sources, follows legal-specific instructions, and includes controls meant to protect confidential client information. In plain terms, it's like giving a smart assistant a law library card and a set of firm rules, rather than letting it guess from random web pages.
On OpenAI's own tests — 200 real US legal research questions — Astra for Law passed the overall correctness check on 54% of questions, compared with about 39% for the regular AI using web search. OpenAI calls that a 40% improvement. It also found 24% more relevant cases and up to 54% more useful passages from correct court opinions. But here's the honest part: 54% correct means it missed on nearly half the questions. Any lawyer using it still has to check every case and conclusion by hand.
Rather than replace the software law firms already use, OpenAI is plugging into it. The company launched 26 partner-built connectors linking ChatGPT to tools for document review, billing, and case management, including Relativity, Clio, iManage, Intapp and DeepJudge. Thomson Reuters is connecting its legal products, and legal AI startups Harvey and Legora can build on Astra for Law. Law firms Sullivan & Cromwell, Ropes & Gray and Cooley have already built custom tools on top of it.
The rollout is limited for now: selected US law firms only, through OpenAI's Trusted Access program, with an option to keep data from being stored. For everyone outside law, the lesson is bigger than lawyers. This is the template for AI in medicine, accounting and finance — specialized, connected to professional tools, and still requiring a human to double-check the work.
- OpenAI's Astra for Law searches a dedicated US legal library instead of the open web, and can plug into tools lawyers already use, like Clio and Relativity.
- In OpenAI's own tests it was right only 54% of the time — better than the general AI, but still wrong on nearly half of legal questions.
- It's limited to selected US law firms for now, with privacy options for confidential client data, and signals a wider trend of AI tailored to specific professions.
Why It Matters
Faster, cheaper legal research could lower your legal bills, but AI mistakes mean humans must still verify everything.