New AF Method Outperforms Phoneme Tokens for Unseen Languages
Continuous articulatory features beat discrete phoneme labels on rare sounds.
Recent Phonetic Foundation Models (PFMs) for Speech-to-IPA transcription rely on grapheme-to-phoneme (G2P) labels—but those labels often aren't phonetically faithful. In a new paper accepted at Interspeech 2026, Ryo Magoshi and colleagues from Kyoto University show that a PFM trained on G2P-labeled data from many languages still fails at zero-shot phonetic classification on unseen sounds like Chinese aspiration contrasts and Japanese moraic nasals. The core issue: multilingual coverage with discrete IPA tokens isn't enough for truly novel phonetic distinctions.
To overcome this, the team proposes classifying phonemes using continuous Articulatory Feature (AF) vectors extracted from every audio frame. These vectors encode articulator positions (tongue, lips, etc.) in a language-agnostic space. The AF-based approach outperforms discrete token methods across both tasks, with the biggest gains on rare phones. Crucially, the optimal temporal aggregation strategy depends on the distinction: single-frame classification works best for aspiration, while segmental (multi-frame) aggregation substantially improves nasal classification. The work suggests a path toward more linguistically universal speech recognition without needing exhaustive IPA token coverage.
- PFMs trained on G2P labels fail on zero-shot tasks like Chinese aspiration and Japanese moraic nasals.
- Proposed method uses continuous Articulatory Feature vectors per frame, outperforming discrete IPA tokens.
- Optimal temporal aggregation differs: single-frame for aspiration, segmental for nasals (accepted at Interspeech 2026).
Why It Matters
Robust zero-shot phonetic classification enables accurate speech processing for low-resource and unseen languages without labeled data.