Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

PayPal researchers have published an empirical study demonstrating how NVIDIA's EAGLE3 speculative decoding technique dramatically accelerates their Commerce Agent AI system. The agent, powered by a fine-tuned Llama 3.1-Nemotron-Nano-8B-v1 model, was benchmarked across 40 configurations using vLLM on 2xH100 hardware. The team tested different speculative token counts (gamma=3 and gamma=5), concurrency levels (1-32), and sampling temperatures (0, 0.5) to find optimal performance.

Key findings show that gamma=3 configuration achieved 22-49% throughput improvement and 18-33% latency reduction with zero additional hardware cost. Acceptance rates remained remarkably stable at approximately 35.5% for gamma=3 across all conditions, while gamma=5 yielded diminishing returns with only 25% acceptance rate. Most impressively, the study found that speculative decoding on a single H100 GPU matches or exceeds NVIDIA NIM's performance on two H100s, enabling 50% GPU cost reduction.

The research builds on PayPal's prior NEMO-4-PAYPAL work that reduced latency and cost through domain-specific fine-tuning. LLM-as-Judge evaluation confirmed that output quality was fully preserved despite the significant speed improvements. This represents a major optimization for PayPal's commerce applications, where faster response times directly impact user experience and transaction completion rates while reducing infrastructure costs.