LLM-based uncertainty assessment of social media situational signals for crisis reporting

A new paper from arXiv (2605.00829) introduces an LLM-based framework that explicitly accounts for uncertainty when extracting situational awareness from social media during crises. Current automated approaches treat all social media posts as equally credible, but Douglas et al. add an uncertainty assessment layer that evaluates whether individual claims plausibly reflect real-world conditions. By conditioning on external proxy data—specifically USGS PAGER earthquake impact summaries—the system elicits confidence scores for each claim. These scores are then used to generate crisis reports that communicate not only what is being reported, but the level of certainty behind each piece of information.

Tested on over 200,000 earthquake-related Twitter/X posts, the framework demonstrates how integrating external authoritative data can filter and prioritize social media signals. The authors argue this uncertainty-aware design supports human crisis communicators under time pressure, and provides a reusable architecture for any LLM-based situational awareness pipeline. The approach is particularly relevant for large-scale emergencies where misinformation and noise proliferate, offering a data-driven way to flag high-confidence reports for immediate action while flagging low-confidence ones for verification.