Gefen achieves 8x memory reduction vs AdamW by compressing optimizer states with low-rank approximations and quantization?

Gefen achieves 8x memory reduction vs AdamW by compressing optimizer states with low-rank approximations and quantization

swap one line of code in PyTorch; no hyperparameter changes needed

Paper and open-source implementation available on arXiv and GitHub for immediate experimentation?

Paper and open-source implementation available on arXiv and GitHub for immediate experimentation

Open Source

r/LocalLLaMA June 25, 2026

⚡New optimizer reduces GPU memory by 8x without sacrificing model accuracy or convergence speed.

Deep Dive

Researchers have introduced a new optimizer called Gefen, with a paper on arXiv and an open-source GitHub repository.

Key Points

Gefen achieves 8x memory reduction vs AdamW by compressing optimizer states with low-rank approximations and quantization
Drop-in replacement: swap one line of code in PyTorch; no hyperparameter changes needed
Paper and open-source implementation available on arXiv and GitHub for immediate experimentation

Enables training massive models on limited hardware, democratizing AI research and reducing cloud costs significantly.