DeepSeek V4 Flash on a single AMD MI300X

The MI300X features 192 GB of HBM3 memory and 5.3 TB/s bandwidth—2.4× more capacity than an H100 SXM5. The entire 304B model fits in memory without PCIe streaming, leaving room for GPU KV cache and CPU offload of evicted cache entries. It also costs roughly half as much at list price.

MI300X uses the AMD/Graphcore E4M3 FP8 variant while newer GPUs use OCP-standard FP8, requiring compatibility fixes. Additional challenges included missing AITER fast paths on gfx942, HIP-graph hazards in sparse MLA decode, and MoE routing bugs at high concurrency. The repository includes pre-packaged fixes.

Single-stream decode: 168.6 tok/s, prefill: 7.9–8.5K tok/s, 8 concurrent streams: 542 tok/s aggregate, 64-stream burst without OOM. The model validates 256K context (architecture supports 1M).

It provides correctness overlays for MI300X compatibility, a validated serving configuration with DSpark-7 speculative decoding, AITER GEMM tuning tables for gfx942, and a hybrid KV strategy combining 20 GB GPU cache with 96 GiB CPU offload. The official vLLM recipe targets NVIDIA and newer AMD GPUs, not single-MI300X deployments.