Activation Steering Boosts Synthetic Data for Low-Resource Languages

A new preprint from Jan Cegin, Daniil Gurgurov, Yusser Al Ghussin, and Simon Ostermann proposes activation steering for generating synthetic data in low-resource languages. Instead of relying on few-shot prompting—which increases inference costs and can reduce lexical diversity—the researchers manipulate internal LLM representations to enforce linguistic identity (Language Steering) or well-formedness (Quality Steering). Language Steering shifts activations toward a target language's region in the model's latent space, while Quality Steering contrasts human-written and backtranslated text to capture naturalness. The methods were evaluated on 4 open-source LLMs (including Llama, Mistral, and others) across 11 typologically diverse languages (e.g., Swahili, Tamil, Welsh) for sentiment analysis and topic classification tasks. Fine-tuned smaller classifiers trained on steered data often matched or surpassed those trained on few-shot-generated data, especially for languages with very few training examples.

The study found that steering applied at early transformer layers yielded the largest gains in data diversity and downstream performance. In zero-shot settings, activation steering produced more varied outputs than few-shot baselines, reducing lexical anchoring—a common problem when models copy phrasing from provided examples. Quality Steering, in particular, helped generate more grammatically diverse and context-appropriate sentences. The paper suggests activation steering can be a cheaper, more effective alternative for low-resource synthetic data generation, as it requires no additional examples at inference time. This opens the door to better NLP tools for underserved languages without heavy computational overhead.