LLMs beat specialized models on multimodal clinical prediction without fusion architectures

Researchers from DFKI and Charité propose a radical simplification for clinical prediction: instead of using dedicated encoders for text, vitals, and labs with complex fusion layers, they convert all patient data into a single natural language string and fine-tune a pretrained large language model end-to-end. They tested this on three diverse tasks — in-hospital mortality using MIMIC-III, graft failure from longitudinal German transplant data, and emergency triage classification from ambulance records. They compared both encoder-based (ModernBERT) and decoder-based models (Llama 3.1, Gemma, DeepSeek-R1-Qwen, Qwen3) against established multimodal baselines and, for graft failure, a gradient boosting model currently used in clinical practice.

The results show that unified textual serialization matches or exceeds all task-specific multimodal baselines across the three tasks. Notably, it outperformed the gradient boosting system on graft failure prediction, the current clinical standard. The approach requires no specialized fusion architecture — just fine-tuning a single LLM on serialized patient records. This substantially reduces system complexity while achieving state-of-the-art performance, suggesting that LLMs can act as universal clinical predictors by simply reading patient data as text. The findings could accelerate deployment of AI in hospitals by eliminating the need to design custom model architectures for each clinical task.