Karpathy: AI Training Costs Drop 40% Per Year — How Deflation Is Reshaping the Industry

Overview

Andrej Karpathy revealed a striking finding through his nanochat project. In 2019, it cost OpenAI approximately $43,000 to train GPT-2 (1.5B parameters). In 2026, achieving the same performance costs just $73 — a roughly 600× cost reduction, demonstrating an annual deflation rate of approximately 40%.

This article examines the structural factors behind AI training cost deflation and its implications for the industry, based on Karpathy’s analysis.

The Evolution of GPT-2 Training Costs

2019: $43,000

• Hardware: 32 TPU v3 chips (256 TPU v3 cores)
• Training time: ~1 week (~168 hours)
• Cloud cost: $8/hour per TPU v3 × 32 × 168 = $43,000

2026: $73

• Hardware: Single 8×H100 GPU node
• Training time: ~3 hours
• Cloud cost: ~$24/hour × 3 = $73

Cost Trajectory (GPT-2 equivalent performance)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2019:  $43,000  ████████████████████████████████
2020:  $17,200  █████████████
2021:   $6,880  █████
2022:   $2,752  ███
2023:   $1,101  ██
2024:     $440  █
2025:     $176  ▏
2026:      $73  ▏
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Four Structural Drivers of Cost Decline

Karpathy attributes the decline not to a single factor but to simultaneous improvements across four axes.

1. Hardware Evolution

The transition from TPU v3 to H100 represents a fundamental leap in computational efficiency.

• FP8 compute support: Lower training precision while maintaining quality
• HBM3 memory: 3TB/s bandwidth eliminates memory bottlenecks
• NVLink 4.0: 900GB/s inter-GPU communication maximizes multi-GPU efficiency

2. Software Optimization

Software stack improvements deliver dramatic performance gains on identical hardware.

• Flash Attention 3: ~9% tokens/sec improvement with native tensor layout, unifying training and inference
• torch.compile: JIT compilation removes Python overhead
• Sliding Window Attention: SSSL pattern (3 short-window + 1 long-window layers) reduces compute without quality loss

3. Algorithm Innovation

Optimizer and architecture innovations fundamentally improve training efficiency.

• Muon optimizer: Polar Express orthogonalization, NorMuon variance reduction, cautious weight decay
• Per-layer residual scalars: x = λ_resid * x + λ_x0 * x0 yields 0.003-0.01 bpb improvement across all model sizes
• Value Embeddings: Applied at alternating layers, adding ~150M parameters at near-zero FLOPs
• ReLU² activation: Sparse and cheaper than GELU

4. Data Pipeline Optimization

High-quality data curation and efficient loading increase training efficiency.

• FineWeb-edu: Curated educational web data maximizes data efficiency
• BOS-aligned dataloader: Every sequence starts with BOS token, eliminating the need for midtraining
• BestFit-Crop packing: 100% utilization, ~35% waste reduction compared to naive cropping

What Didn’t Work

Karpathy transparently shared techniques that failed, providing valuable insights for the community.

Technique	Result
Multi-token prediction (MTP)	+13GB memory, no improvement
FP8 for lm_head	Works but +2GB memory, only 1% speedup
Half-truncated RoPE	No improvement
Skip connections / backout	No improvement, +2GB memory
Bigram embeddings (Engram-lite)	Works but insufficient benefit for added complexity

Industry Impact

Collapsing Entry Barriers

A 40% annual cost decline accelerates the democratization of AI training. Training at scale, once exclusive to big tech, is now accessible to startups and individual researchers.

Shifting Competitive Axes

As cost ceases to be a differentiator, competition shifts to:

• Data quality: Securing superior training data
• Fine-tuning expertise: Domain-specific optimization capabilities
• Inference efficiency: Serving costs matter more than training costs

Strengthening the Open-Source Ecosystem

Training a GPT-2-class model for under $100 means the open-source community can experiment and innovate at unprecedented speed. nanochat itself comprises roughly 1,000 lines of code, making it highly educational.

Outpacing Moore’s Law

The 40% annual decline outpaces Moore’s Law (~29% annual cost reduction). This results from the compound effect of simultaneous improvements in hardware, software, algorithms, and data.

Conclusion

Karpathy’s nanochat project goes beyond a mere benchmark record — it empirically demonstrates the structural deflation of AI training costs. The simultaneous improvement across four axes — hardware, software, algorithms, and data — produces a remarkable 40% annual decline, fundamentally reshaping the competitive landscape of the AI industry.

Notably, Karpathy himself states this is “an underestimate and that further improvements are still quite possible.” The deflation isn’t over yet.

References

• Beating GPT-2 for <$100: the nanochat journey — Karpathy
• Reddit r/LocalLLaMA Discussion
• nanochat GitHub Repository