Cognitive Alignment Deciphered: A Self-Developed Scenario-Based Prompt Scale Coupled with Representational Similarity Analysis and Social Network Analysis for Unraveling Bias Mechanisms Across Humans and LLMs
AI models show rigid bias networks vs. human flexibility, but prompt fixes work
A new study from researcher Chengrui Zhou introduces the Cognitive Bias Assessment Scale (CBAS), a scenario-driven prompt template that systematically measures 58 cognitive biases across five dual-system dimensions: Calculation, Belief, Information, Social, and Memory. Unlike traditional self-report tools, CBAS uses contextual scenarios to improve ecological validity. Psychometric testing with 330 participants confirmed reliability (Cronbach's alpha 0.714) and good model fit (RMSEA 0.057, CFI 0.908), validating the scale for human-AI comparison.
Applying Representational Similarity Analysis (RSA) and Social Network Analysis (SNA), the study compared human age groups with three large language models: Baidu ERNIE 3.5 8K, DeepSeek V3, and DeepSeek R1. Results showed humans maintain coherent 'hot-cold' integration with high inter-individual variability, while LLMs display fragmented, inflexible response patterns and lower variability. Human cognitive networks exhibit strong inter-module connectivity, whereas LLMs show fixed core biases and isolated information processing components. Crucially, prompt interventions combining role-playing and bias mitigation instructions significantly improved LLM accuracy, reaching 84.86% for DeepSeek R1 and 78.24% for DeepSeek V3, partially reshaping their internal bias representations.
- CBAS covers 58 cognitive biases across five dimensions (Calculation, Belief, Information, Social, Memory) with strong psychometric validation (Cronbach's alpha 0.714)
- Humans show coherent, variable bias networks with strong inter-module connectivity; LLMs display fragmented, inflexible patterns with fixed core biases
- Prompt interventions improved DeepSeek R1 accuracy to 84.86% and DeepSeek V3 to 78.24%, demonstrating bias mitigation through targeted prompts
Why It Matters
Establishes a replicable pipeline for measuring and mitigating cognitive biases in LLMs, crucial for safer AI deployment.