VTBench: A Multimodal Framework for Time-Series Classification with Chart-Based Representations

Time-series classification (TSC) has traditionally relied on raw numerical inputs, but VTBench, developed by Madhumitha Venkatesan, Xuyang Chen, and Dongyu Liu, introduces a multimodal approach that pairs numerical sequences with lightweight, human-readable charts. The framework generates four chart types — line, area, bar, and scatter — each offering a complementary view of the same signal. Unlike texture-based encodings like Gramian Angular Fields, these charts are intuitive and require minimal preprocessing. VTBench's modular architecture supports single-chart fusion, multi-chart fusion, and full multimodal fusion with raw data, allowing researchers to systematically evaluate which combinations work best.

In experiments across 31 diverse UCR datasets, the team found that chart-only models are surprisingly competitive on smaller datasets, sometimes outperforming pure numerical models. Combining multiple chart types improves accuracy by capturing complementary visual cues, but multimodal fusion (charts + raw numbers) only improved or maintained performance when charts added non-redundant information — otherwise it degraded accuracy. The paper distills practical guidelines for selecting chart types and fusion strategies, providing a unified foundation for interpretable, effective time-series classification. This could be especially useful for domains like finance, healthcare, and IoT where both accuracy and interpretability matter.