Character-trained models fail to generalize from chat to agentic tasks

Character training via distillation and supervised fine-tuning (SFT) reliably shapes model personas on chat benchmarks, but a new replication study by Nathaniel Mitrani shows this behavior collapses under distribution shift. Using three base models (Llama-3.1-8B, Qwen-2.5-7B, Gemma-3-4B) fine-tuned with LoRA adapters for 10 distinct personas (sarcasm, humor, remorse, etc.), Mitrani first reproduces Maiya et al.'s in-distribution results: a ModernBERT classifier achieves macro-F1 scores of 0.86–0.95 on PURE-DOVE chat prompts. However, when the same models are placed in an agentic email scaffold — a multi-turn tool-use loop where the model emits JSON tool_calls and the final email body is scored — the classifier's macro-F1 drops to 0.29–0.55, a 40–60 point decline. The degradation is uneven across personas, with some exhibiting steeper drops than others.

This fragility aligns with prior work by Li et al. and Kutasov et al., showing that alignment policies learned via SFT/DPO over narrow input distributions fail to generalize to agentic contexts. The agentic scaffold here avoids confounding factors by scoring only the email body (free-form prose) rather than tool selection order, isolating the persona expression channel. The results provide modest but clear evidence that character-trained models are brittle when the input format shifts from chat turns to tool-use rollouts. For developers building autonomous agents, this means persona consistency cannot be assumed outside carefully matched dialog formats — practical reliability requires either broader training distributions or alternative alignment methods like rationale-based fine-tuning.