Assessing the Feasibility of a Video-Based Conversational Chatbot Survey for Measuring Perceived Cycling Safety: A Pilot Study in New York City
A new study replaces traditional surveys with an LLM-powered chatbot that analyzes real-time reactions to street videos.
A team of researchers from MIT and other institutions has published a novel pilot study proposing a new method for urban planning data collection. The paper, "Assessing the Feasibility of a Video-Based Conversational Chatbot Survey for Measuring Perceived Cycling Safety," addresses a key flaw in conventional surveys: their reliance on imperfect human recall. By leveraging large language models (LLMs), the team built a conversational AI chatbot that presents participants with videos of real street segments in New York City and engages them in a dialogue about their immediate safety perceptions.
The chatbot was built using a modular LLM architecture, integrating specialized prompt engineering, state management, and rule-based controls to guide the structured human-AI interaction. In the pilot, 16 participants provided complete responses for nine different NYC street segments. The method's feasibility was rated positively, with users giving it a mean score of 5.00 out of 7 for ease of use, supportiveness, and efficiency. The collected qualitative data was then analyzed using Natural Language Processing (NLP) tools like KeyBERT for feature extraction, K-means clustering for semantic analysis of reasons and suggestions, and regression models to estimate the effects of built-environment variables.
The results demonstrate a significant proof-of-concept for applying conversational AI to real-world urban planning challenges. This method captures richer, context-specific data than static questionnaires by grounding responses in a shared visual stimulus. The study concludes that AI chatbots represent a promising new tool for collecting nuanced data on human perception, behavior, and future visions for transportation systems, potentially leading to more responsive and evidence-based city design.
- The system uses a modular LLM chatbot to conduct surveys based on street videos, moving beyond recall-based questions.
- In the NYC pilot, 16 users rated the chatbot experience 5/7 for ease of use and 3.47/5 for usability traits like friendliness.
- Researchers used NLP (KeyBERT) and K-means clustering to analyze responses, extracting built-environment factors affecting safety perceptions.
Why It Matters
This AI-driven method could revolutionize urban planning by providing richer, real-time perceptual data to design safer cities.