AI Could Speed Up Legal Document Searches in Brazil — With a Catch

Searching for legal documents is like finding a needle in a haystack — and in Brazil, it's especially hard because there's little public data to train and test search systems. To change that, researchers created NormasTCU, a new dataset with 14,469 legal documents from the Brazilian Federal Court of Accounts, 46 search queries, and over 3,000 human judgments on which results were relevant. This gives developers a way to check whether search tools actually work for Portuguese legal texts.

The team then tested something practical: Can AI do the job of the human judges who decide if a document is relevant? They asked three large language models to grade 812 document-query pairs using two different prompting styles. The idea was to see if AI could replace expensive human reviewers, who are slow and costly to hire for specialized collections.

The results were mixed. AI was too optimistic — it consistently rated documents as more relevant than humans did. Agreement with human judgments was only fair to moderate. However, when the goal was to compare different search systems, AI-generated rankings were often very similar to human rankings, especially when measuring how good the top 10 results are. In fact, AI rankings sometimes matched the human reference better than individual human annotators did. But AI was less reliable when measuring precision and recall — that is, how many of the relevant results you actually find.

What does this mean for everyday people? If you work with legal or government documents in Portuguese, this could lead to better search tools at lower cost. Developers could use AI to quickly test and improve search systems without hiring dozens of human reviewers. But the study warns against using AI when missing a single relevant document is unacceptable — like in legal investigations or audits. AI can be a smart assistant, but for high-stakes searches, a human still needs to be in the loop.