LLM Chatbots vs Dashboards: Study Finds Conversational AI Best for Simple Tasks Only

Researchers from the University of Pisa and Cognetive Labs conducted a 2x3 mixed factorial experiment with 134 industrial decision-makers to compare an LLM-based conversational user interface (CUI) against a traditional graphical dashboard for manufacturing decision support. Participants completed three tasks of increasing complexity while researchers measured mental workload (MWL), decision accuracy, completion time, and intended reliance. Self-reported data literacy was tested as a potential moderator.

Results showed that the CUI significantly reduced perceived MWL overall and enabled faster completion for less demanding tasks. However, as task complexity increased, both the workload and speed advantages diminished. Neither interface produced a consistent edge in decision accuracy across all task types. Importantly, participants indicated they would not rely solely on the CUI for future decisions, preferring the inspectable, persistent visual representation of dashboards for complex scenarios. Data literacy did not reliably moderate any interface effects, suggesting broader applicability of the findings.