AC-GPT lets LLMs handle any conditional in one forward pass
New method fine-tunes existing models for past, future, and mixed context sampling
AC-GPT introduces a straightforward tweak to standard causal Transformers, enabling them to evaluate and sample from arbitrary conditionals—such as text blocks conditioned on past and future tokens—all in a single forward pass. This is a leap over prior methods that often degraded generation quality or required complex architectural changes. The key insight is that AC-GPT preserves the left-to-right autoregressive ordering and the next-token prediction objective, the very mechanisms that make modern LLMs efficient at training and strong at standard language modeling.
Crucially, this compatibility means any pre-trained causal LLM can be fine-tuned for arbitrary conditioning without sacrificing its core performance. In experiments, AC-GPT outperformed baseline approaches on modeling arbitrary conditionals while showing no drop in standard left-to-right metrics. This opens up practical applications like infilling, text editing, and constrained generation, all using familiar transformer architectures.
- AC-GPT modifies causal Transformers to handle arbitrary conditionals (past, future, mixed) in one forward pass.
- Method preserves standard left-to-right ordering and next-token prediction, enabling fine-tuning of existing LLMs.
- Empirical results show AC-GPT outperforms prior arbitrary-conditioning baselines without degrading standard performance.
Why It Matters
Enables flexible text infilling and editing on existing LLMs without architectural overhauls or performance trade-offs.