Gloss-free AI pretraining doubles cross-dataset sign spotting accuracy

Sign-language AI typically depends on expensive gloss annotations — linguistic labels that mark word boundaries and meanings. A team of Turkish computer scientists (Oğuz Akif Tüfekcioğlu, Ezgi Ekin, Mustafa Kaan Çevik, and Hacer Yalim Keles) from Ankara University presents a gloss-free alternative in their new paper, accepted at the LIMIT Workshop at ECCV 2026. Instead of manual labels, they use broadcast news with spoken-language transcripts as weak supervision, exploiting the loose alignment between audio and signing to train a reusable sign encoder.

They created TSL-News, a Turkish broadcast corpus, and derived pseudo-gloss labels directly from transcripts. To handle Turkish's morphologically rich structure, they compared rule-based lemmatization against a constrained LLM-assisted normalization method over a fixed vocabulary. The LLM approach proved significantly better at producing consistent pseudo-gloss targets, reducing fragmentation that weakens representation learning.

In cross-dataset evaluation on their new TSL Spotting Benchmark (built from the TSL Dictionary corpus), the LLM-assisted encoder raised top-5 temporal localization mean IoU from 0.235 to 0.465 — a ~2x jump — with 56.2% of examples reaching an IoU of at least 0.50. A frequency analysis confirmed the gains weren't just from memorizing frequent labels. The same pretrained encoder also improved downstream translation, lifting BLEU-4 from 9.60 to 11.04 and ROUGE from 23.48 to 27.43.

This work demonstrates that loosely aligned broadcast data can effectively pretrain sign representations that capture lexical content and temporal structure, without glosses. For low-resource sign languages like Turkish Sign Language (TSL), this could dramatically lower the barrier to building robust recognition and translation systems, since publicly available TV broadcasts can serve as training data.