H200 vs MI300X: What actually matters for training
Jun 28, 2026·Priya Ramesh
benchmarks
gpu
training
The short version
On dense transformer pretraining at the 70B parameter class, H200 still wins on tokens per dollar when you can secure it. MI300X is competitive on memory-bound workloads and often available when H200 isn't.
Benchmarks we ran
| Workload | H200 (tok/s/GPU) | MI300X (tok/s/GPU) |
|---|---|---|
| Llama-3 70B pretrain | 3,140 | 2,610 |
| Mixtral 8x22B SFT | 1,890 | 2,050 |
| Long-context (128k) inference | 42 req/s | 51 req/s |
When to pick MI300X
- Long context inference where 192GB HBM3e matters.
- MoE workloads with high expert count.
- When H200 lead time is > 3 weeks.
When to pick H200
- Dense pretraining and standard SFT.
- NCCL-heavy multi-node runs.
- Anywhere the ecosystem tax on ROCm still bites you.