[Release] Apex-1: A 350M Tiny-LLM trained locally on an RTX 5060 Ti 16GB

Independent developer LH-Tech-AI has launched Apex-1, a 350 million parameter 'Tiny-LLM' designed to bring capable language AI to resource-constrained environments. The model was trained entirely locally on a single consumer-grade NVIDIA RTX 5060 Ti GPU with 16GB of VRAM, showcasing the potential of accessible hardware for model development. Its architecture is based on nanoGPT/Transformer, and it was pre-trained on a 10 billion token subset of the high-quality FineWeb-Edu dataset to instill reasoning and world knowledge.

For improved usability, Apex-1 was then fine-tuned on the Alpaca-Cleaned dataset for better instruction following. The developer provides the model weights in both standard PyTorch and ONNX formats, the latter being optimized for deployment in mobile and web applications. The model is positioned for tasks like basic summarization and simple question-answering on hardware traditionally unable to run larger LLMs. LH-Tech-AI has already announced that an enhanced Apex 1.5 and a dedicated code-specialized version are in development.