Intuition First or Reflection Before Judgment? The Impact of Evaluation Sequence on Consumer Ratings
Research shows asking for a star rating before a written review creates more extreme 1-star and 5-star scores.
A new study titled 'Intuition First or Reflection Before Judgment? The Impact of Evaluation Sequence on Consumer Ratings' provides a crucial look at how the design of review interfaces fundamentally shapes the feedback we see online. Led by researchers He Wang, Yueheng Wang, Ziyu Zhou, and Hanxiang Liu, the paper investigates two common sequences: 'Rating-First' (assign a star score, then write) and 'Review-First' (write text, then assign stars). Through controlled experiments and analysis of real-world data from Yelp (Rating-First) and Letterboxd (Review-First), the team discovered that the order of operations triggers different psychological processes, with significant consequences for the resulting data.
The core finding is a polarization effect driven by 'affective heuristics.' When users are prompted to give a star rating first, they rely more on their immediate gut feeling. In high-quality service contexts, this leads to higher ratings; in low-quality contexts, it leads to significantly lower ratings. This creates the classic 'bimodal' distribution of mostly 1-star and 5-star reviews seen on platforms like Yelp. Conversely, the 'Review-First' approach, used by platforms like Letterboxd, forces users to articulate their thoughts in writing first. This cognitive effort moderates the initial emotional response, leading to more concentrated, middling ratings and a less polarized distribution.
The research also identifies key moderators. The polarization effect is amplified for hedonic products (like entertainment or luxury items) compared to utilitarian ones. The mechanism is explained as a serial mediation path: the Rating-First sequence strengthens the influence of affective heuristics while reducing subsequent cognitive effort during review writing. This work moves beyond simply observing review patterns to explaining the cognitive architecture behind them, offering a powerful lens for both academics and platform designers to understand rating authenticity.
- Rating-First interfaces (like Yelp) create polarized, bimodal score distributions by triggering affective gut reactions.
- Review-First interfaces (like Letterboxd) yield more moderate, concentrated ratings by prompting cognitive reflection through writing first.
- The effect is stronger for hedonic products (e.g., movies, restaurants) than for utilitarian ones (e.g., tools, appliances).
Why It Matters
Platforms can engineer more authentic and useful review ecosystems by strategically designing their feedback interface sequence.