STaR-DRO: Stateful Tsallis Reweighting for Group-Robust Structured Prediction

A research team led by Samah Fodeh has introduced STaR-DRO, a novel framework designed to make large language models (LLMs) like Llama 3.3 more accurate and reliable for complex, structured prediction tasks, particularly in high-stakes fields like healthcare. The framework has two core components. First, a task-agnostic prompting strategy uses XML-based instructions, disambiguation rules, and self-validation to combat common LLM failures like format drift and evidence hallucination during structured generation. Second, and most innovatively, is the STaR-DRO (Stateful Tsallis Reweighting for Distributionally Robust Optimization) training method. This technique smartly reweights the model's focus during fine-tuning, concentrating learning effort only on persistently difficult data groups—such as rare but clinically critical communication categories—while ignoring easier or volatile examples.

The team rigorously evaluated their combined framework on EPPC Miner, a benchmark for extracting hierarchical medical codes and evidence spans from secure patient-provider messages. The prompting strategy alone provided a significant +15.44 average F1 score boost in zero-shot settings across four Llama models. Building on supervised fine-tuning, the STaR-DRO optimization delivered further critical gains on the hardest semantic decisions. For the powerful Llama-3.3-70B-Instruct model, Code F1 rose from 79.24 to 81.47 and Sub-code F1 from 67.78 to 69.30. Most importantly, it reduced group-wise validation cross-entropy by up to 29.6% on the most difficult clinical categories. This isn't just a statistical win; these 'hard groups' often correspond to rare but consequential communication behaviors, meaning the improvement directly translates to more trustworthy AI tools for patient-centered care analysis and clinical communication mining.