New MLLM system boosts forensic image retrieval for tattoos and face sketches

A team of researchers led by Ricardo González-Gazapo has introduced a unified retrieval framework designed to bridge the modality gap in forensic image analysis. The system tackles four critical forensic tasks: matching a tattoo from a query image, retrieving tattoos based on verbal witness descriptions, finding tattoos from hand-drawn sketches, and identifying faces from forensic sketches. By leveraging a multimodal large language model (MLLM), the framework automatically generates structured textual descriptions for both query images and gallery images. These descriptions are then embedded using sentence-transformer models for text-based comparison, while visual features are extracted using state-of-the-art encoders tailored to each task.

The key innovation lies in a fusion strategy that combines text- and image-based similarity scores. Across experiments, the multimodal fusion consistently improved retrieval precision and robustness compared to using either modality alone. This advantage is most pronounced in challenging real-world scenarios—such as partial tattoos, rough sketches, or incomplete witness statements—where visual information is limited or noisy. The paper demonstrates how modern MLLMs can operationalize tasks that traditionally relied on manual expert analysis, offering a scalable and automated pipeline for forensic investigations involving tattoos, facial composites, and witness descriptions.