ASK+ Boosts SLM-Guided RL Agents by 15% with Smarter Prompts and Chain-of-Thought
New ASK+ technique helps small language models guide RL agents in the dark, boosting success rates up to 17%.
Reinforcement learning (RL) agents often operate under partial observability, meaning they only see part of the environment. Researchers attempted to correct their actions using small language models (SLMs) via uncertainty-gated assistance (ASK). However, the vanilla approach almost never overwrote the agent's actions because the SLM was given a bare egocentric prompt with insufficient context — a context problem, not a capacity problem.
To fix this, the team introduced ASK+, which supplies the SLM with trajectory-aware context, including a partially revealed map, visited positions, and action history, plus structured chain-of-thought reasoning. This transformed the SLM from a passive redundancy check into an informative consultant. Predictive entropy signals measured action uncertainty, not state uncertainty, and remained valid in POMDPs. Results show dramatic gains: on DoorKey, ASK+ hit 93% success (vs. 89% for vanilla and PPO); on FourRooms, 70% (up from 53%); on HigherLower, 73.7% (matching the SLM-only upper bound). Notably, Qwen3.5-2B matched or exceeded Qwen3.5-4B, confirming that prompt engineering and selective gating dominate the impact of model scale.
- Vanilla ASK under partial observability had near-zero overwrite rate because the bare prompt lacked trajectory context.
- ASK+ provides trajectory-aware context (partial map, visited positions, action history) and structured chain-of-thought reasoning.
- Results: DoorKey 93% (+4%), FourRooms 70% (+17%), HigherLower 73.7%, with Qwen3.5-2B matching Qwen3.5-4B.
Why It Matters
Proves that prompt engineering and selective gating can make small models as effective as larger ones for real-time RL guidance.