Trained a Qwen2.5-0.5B-Instruct bf16 model on Reddit post summarization task with GRPO written from scratch in PyTorch - updates! [P]

An independent developer has successfully fine-tuned Alibaba's Qwen2.5-0.5B-Instruct model for the specific task of summarizing Reddit posts. The project's core innovation is the implementation of a custom Group Relative Policy Optimization (GRPO) algorithm from scratch in PyTorch, a reinforcement learning technique used to align model outputs with human preferences. The training infrastructure was notably built on a budget-friendly cluster of three Apple Mac Minis running MLX, Apple's machine learning framework for its silicon. In this distributed setup, one node drove the GRPO training process, while the other two handled inference and generated text "rollouts" using the vLLM serving library.

The training employed a dual-reward system to shape the model's behavior. A length penalty reward encouraged outputs close to a target length, while a quality reward—using metrics like ROUGE-L—ensured the summaries maintained structural similarity to high-quality reference examples. The developer trained two model variants: one with only the length penalty and another combining it with the quality reward. For evaluation, they used an "LLM-as-a-Judge" approach, leveraging OpenAI's GPT-5 through the DeepEval framework to score generated summaries on four critical axes: Faithfulness (avoiding hallucinations), Coverage (capturing key points), Conciseness, and Clarity. The initial successful run yielded an average output length of 64 tokens, demonstrating the feasibility of sophisticated RLHF-style training on accessible, consumer-grade hardware.