Nova Forge SDK series part 2: Practical guide to fine-tune Nova models using data mixing capabilities
New data mixing technique preserves 90% of MMLU scores while boosting domain performance by 12 F1 points.
Amazon has released the second part of its Nova Forge SDK series, providing a hands-on guide to fine-tuning Nova AI models using a novel data mixing technique. This approach addresses a critical challenge in AI customization: how to adapt models to specific domains without destroying their general capabilities. The guide demonstrates that blending customer data with Amazon-curated datasets preserves near-baseline Massive Multitask Language Understanding (MMLU) scores while delivering a 12-point F1 improvement on complex Voice of Customer classification tasks spanning 1,420 categories. By contrast, fine-tuning open-source models on customer data alone caused near-total loss of general capabilities.
The comprehensive workflow walks users through five stages: environment setup with the Nova Forge SDK, data preparation and sanitization, training configuration with SageMaker HyperPod runtime, model training using Low-Rank Adaptation (LoRA), and evaluation against both public benchmarks and domain-specific metrics. The technical implementation requires AWS infrastructure including SageMaker HyperPod clusters with GPU instances (like ml.p5.48xlarge), MLflow for experiment tracking, and proper IAM role configurations. Amazon emphasizes starting with short test runs (max_steps=5) to validate configurations before committing to full training runs on these high-end GPU instances.
This release builds on the SDK introduction from Part 1, providing enterprises with a repeatable playbook for AI customization. The data mixing capability represents a significant advancement over traditional fine-tuning approaches, enabling organizations to create specialized AI assistants that maintain broad knowledge while excelling at specific business tasks. The complete code examples and installation scripts make this accessible to teams with AWS infrastructure and Nova Forge access.
- Data mixing preserves 90% of baseline MMLU scores while boosting domain performance by 12 F1 points
- Complete workflow from data prep to evaluation using SageMaker HyperPod with ml.p5.48xlarge GPU instances
- Uses Low-Rank Adaptation (LoRA) for efficient fine-tuning with Amazon-curated datasets blended with customer data
Why It Matters
Enables enterprises to create specialized AI that maintains general knowledge while excelling at specific business tasks.