AI Researchers Want to Make Survey Questions Smarter
Your opinion could shape better AI, but current surveys are clunky and boring.
Human-in-the-loop optimization refers to systems that present choices to a human decision-maker and query their judgment to optimize system parameters iteratively. The user interface typically provides either an absolute rating scale or asks for a preferred option.
However, these approaches can potentially suffer from calibration problems, inconsistencies, and completeness violations of user preferences in such a sequential evaluation. This paper revisits the design space of opinion measurement interfaces for human-in-the-loop optimization.
We conducted a study (N=360) using text summarization and image color enhancement tasks to evaluate human sequential feedback in six representative opinion measurement interfaces. Based on our analysis, we recommend using a listwise approach to support sequential opinion measurement, which counterweighs individual rating and preferential choice limitations and conveys more information for subsequent use. We further discuss the trade-offs in different user interface designs and provide guidelines to inform future research.
- Current surveys for AI feedback are often boring and don't capture what people really think.
- New research suggests interactive interfaces that adapt to your answers could gather better feedback.
- This could lead to AI that understands you better, but it's still in early research stages.
Why It Matters
Better feedback methods could make future AI more personalized and less frustrating for everyone.