Accelerating LLM fine-tuning with unstructured data using SageMaker Unified Studio and S3

AWS has launched a deep integration between Amazon SageMaker Unified Studio and Amazon S3 general purpose buckets, creating a unified pipeline for fine-tuning large vision-language models (VLMs) like Meta's Llama 3.2 11B Vision Instruct. The new workflow eliminates traditional data silos by allowing teams to directly discover, catalog, and process unstructured data (e.g., images, documents) stored in S3 for machine learning. In a detailed demonstration, AWS fine-tuned the Llama model on the Hugging Face DocVQA dataset—containing 39,500 training examples—to improve its visual question answering (VQA) performance for tasks like extracting dates from receipts. The base model scored 85.3% on the Average Normalized Levenshtein Similarity (ANLS) metric, and fine-tuning with varying dataset sizes (1k, 5k, 10k images) aimed to push accuracy higher.

The architecture is built within SageMaker Unified Studio, using separate projects for data producers (who catalog datasets) and data consumers (who train models). It automates the entire ML lifecycle: data ingestion from S3, preprocessing, distributed training on p4de.24xlarge instances, and evaluation using a serverless MLflow tracking server. This managed approach significantly accelerates experimentation by reducing manual data engineering and infrastructure setup. The result is a repeatable, scalable template for enterprises to customize state-of-the-art models like Llama 3.2 Vision for domain-specific document understanding, all within AWS's integrated AI platform.