From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

A research team from Russia has published a significant paper detailing a new multimodal AI framework that substantially improves automated animal identification. The core innovation is moving beyond purely visual systems by augmenting them with synthetic textual descriptions of the animals, creating semantic 'identity priors.' To support this research, the team constructed one of the largest datasets of its kind, containing 1.9 million photographs covering nearly 700,000 unique animals. Their systematic ablation studies identified SigLIP2-Giant as the optimal vision backbone and E5-Small-v2 as the best text encoder for the task.

The proposed system uses a gated fusion mechanism to intelligently combine the visual and text modalities, outperforming simpler methods like concatenation. On comprehensive testing, the model achieved a Top-1 accuracy of 84.28% and an impressively low Equal Error Rate (EER) of 0.0422. This represents an 11% absolute improvement over leading unimodal (vision-only) baselines. The results conclusively demonstrate that integrating synthesized semantic descriptions—such as 'brown tabby cat with white paws'—significantly refines the AI's decision boundaries. This work, published in the *Journal of Imaging*, provides a clear blueprint for building more robust, real-world pet re-identification systems for shelters and lost pet services.