New AI Finds the Sign Language Word You're Trying to Describe
Describe a hand gesture in plain words — get the sign, its meaning, in seconds.
Researchers built SignTrace to solve a longstanding reverse-lookup problem: you remember how a sign moves, but you don't know its meaning or formal feature codes. SignTrace gives you natural-language access to a Chinese sign-language dictionary, combining LLM-based dictionary enrichment, action extraction, dictionary-style rewriting, seven-channel retrieval, and candidate reranking over 6,699 entries. It's been deployed for user trials and received positive informal feedback. On a dictionary-derived benchmark of 500 movement-description queries, it scored 94.0% Hit@1, 97.4% Hit@9, and a mean reciprocal rank of 0.9540 — reranking alone lifted Hit@1 from 71.8% to 94.0%. Median query-processing time is 13.37 seconds with six concurrent queries. The paper itself notes a limit: because benchmark wording and selection come from the dictionary, generalization to descriptions independently produced by users is constrained.
- SignTrace lets you describe a hand gesture in everyday words and returns the matching sign plus its meaning — no need to know formal sign-notation codes.
- It searches a 6,699-entry Chinese Sign Language dictionary and gets the right answer first 94% of the time, up from 71.8% before a final re-ranking step.
- Each search takes about 13 seconds, and the team has already run user trials — though testing used dictionary-style descriptions, not the looser wording real people use.
Why It Matters
Anyone learning sign language, or interpreting for a deaf friend, can now identify an unknown sign in seconds instead of giving up.