NY Judge Warns Lawyers After AI Hallucinates Two Fake Court Cases

A New York Supreme Court judge has issued a sharp rebuke after a lawyer submitted two entirely fabricated court cases generated by artificial intelligence in a lawsuit involving student protesters. Justice Lyle Frank, presiding over the case, called the incident a "cautionary tale" about the unchecked use of generative AI in legal proceedings. The lawyer, representing a group of student demonstrators, had cited the fake precedents—complete with fake case names, docket numbers, and court rulings—in a motion brief. The opposing counsel discovered the cases did not exist, prompting the judge to demand an explanation.

Justice Frank noted that while AI can be a valuable research tool, it is not a substitute for professional diligence. He warned that attorneys must "verify any and all AI-generated output to ensure accuracy" and that reliance on such tools without human oversight could violate ethical rules. The judge stopped short of imposing sanctions but said future violations would be treated severely. This case echoes similar incidents in other U.S. courts where lawyers have been penalized for using AI hallucinations, underscoring the urgent need for legal guidelines on AI use in litigation.