Choosing a serving precision is a hardware-generation decision before it is a quality decision. Hopper GPUs (H100, H200) expose native FP8 tensor cores, while Blackwell GPUs (B200, GB200) add NVFP4 hardware that roughly doubles low-precision throughput again. Because model weights, activations, and the KV cache can each be quantized independently, the 4-bit-versus-8-bit question decomposes into several smaller decisions rather than one. This article assembles a decision framework for infrastructure leads weighing GPU generation, model size, checkpoint provenance, and workload profile. It covers the highest-ROI precision per generation, the weight-versus-KV-cache distinction, the safety margin FP8 provides for small models, and how PTQ- versus QAT-produced checkpoints change the calculus. The scope is production LLM serving on Hopper and Blackwell datacenter hardware; format internals and kernel design are addressed only where they alter a serving decision.
Native hardware support, not software tricks, determines whether a lower‑precision format actually improves throughput.
Because the compute units are hard‑wired, the “best‑ROI precision” for a given GPU generation is fixed:
| Generation | Fastest native format | Relative throughput* |
|---|---|---|
| Ampere | INT8 (native) | ~1.5 × FP16 |
| Hopper | FP8 (native) | ~2 × BF16/FP16 |
| Blackwell | NVFP4 (native) | ~2 × FP8 (≈4 × BF16) |
*Values are vendor spec‑sheet TFLOPS; actual gains depend on model size, batch, and driver version.
TFLOPS numbers and the exact 2 × ratio are taken from NVIDIA spec sheets released between 2025 and 2026 and can shift with SKU revisions, driver updates, or sparsity settings. Likewise, the “native” claim hinges on the availability of the Transformer Engine and the appropriate SDK versions; without them a GPU may fall back to emulation. Keeping the software stack aligned with the hardware generation is therefore essential to realize the advertised ROI.
Quantization delivers value through two tightly coupled mechanisms: memory footprint and memory‑bandwidth consumption.
| Precision | Approx. weight size (70 B model) | Relative bandwidth demand* |
|---|---|---|
| BF16 | ~140 GB | 1 × (baseline) |
| FP8 | ~70 GB | ≈½ of BF16 → ≈½ decode time |
| FP4 (NVFP4) | ~35 GB (raw) ≈ 40‑45 GB practical | ≈¼ of BF16 → ≈¼ decode time |
*Decode is memory‑bandwidth‑bound, with an arithmetic intensity of roughly 1 FLOP per byte at batch‑size 1, so halving the bytes per weight translates to a near‑linear reduction in token‑generation latency【https://www.gmicloud.ai/en/blog/fp8-fp4-quantization-inference】.
FP8 vs. BF16 – Real‑world inference on Hopper GPUs (H100 SXM5) shows 1.6 ×‑2.2 × higher token‑generation rates. vLLM reports up to 1.6 × speed‑up for W8A8 kernels, while NVIDIA’s internal H100 measurements reach 2.2 × at large batch sizes. The gain is modest compared to the raw 2 × TFLOPS ratio because attention, KV‑cache handling, and scheduler overhead consume a sizeable share of the budget.
FP4 vs. FP8 – On Blackwell B200 GPUs, FP4 delivers 1.5 ×‑2 × the throughput of FP8 in production workloads. Spheron’s benchmark of a 70 B Llama model shows 12 841 tok/s for FP4 versus 6 972 tok/s for FP8, a 1.84 × improvement【https://ifactoryapp.com/sap-integration/on-prem-ai/fp4-vs-fp8-vs-fp16-llm-inference】. The shortfall from the theoretical 2 × is due to activations, KV‑cache precision, and kernel overhead.
Because decoding is bandwidth‑limited, the cost per token—not raw TFLOPS—is the decisive metric for serving economics. Recent pricing (2025‑2026) on on‑demand cloud instances yields:
| Config | GPU | Precision | Tokens / s (≈) | $/hr (on‑demand) | Cost / M tokens |
|---|---|---|---|---|---|
| H100 SXM5 | FP8 | 3 066 | $2.01 | $0.71 | |
| H100 SXM5 | BF16 | 1 900 | $2.01 | $1.19 | |
| B200 (Blackwell) | FP4 | 12 841 | $6.03 | $0.13 | |
| B200 (Blackwell) | FP8 | 6 972 | $6.03 | $0.24 |
These figures demonstrate that FP8 on Hopper halves the cost per million tokens, while FP4 on Blackwell cuts it further by roughly another 40 %【https://www.spheron.network/blog/fp4-quantization-blackwell-gpu-cost】.
Reducing weight memory frees VRAM for the KV cache, directly raising the number of concurrent generations a GPU can sustain. For a 70 B model on an 80 GB H100:
Higher concurrency translates to higher aggregate token throughput and lower overall GPU count, reinforcing the cost advantage of lower‑precision formats.
When sizing a serving cluster, compute cost‑per‑token (derived from $/hr, tokens / s, and observed concurrency) rather than theoretical FLOPS. Use the following decision ladder:
All numbers are time‑sensitive (spot vs. on‑demand rates, hardware revisions) and should be re‑derived for each deployment. The core economic insight remains: quantization’s primary dividend is the reduction of memory traffic, which, in a bandwidth‑bound decode phase, translates directly into faster token generation and lower cost per token.
LLM serving workloads face two distinct memory pressures: the permanent weight footprint that lives on the GPU for every forward pass, and the ephemeral KV‑cache that grows linearly with context length and batch size. Treating “4‑bit vs 8‑bit” as a single choice conflates these axes and can lead to sub‑optimal trade‑offs.
Weight quantization
KV‑cache quantization
| Scenario | Recommended weight precision | Recommended KV‑cache precision | Rationale |
|---|---|---|---|
| Short‑context chat (≤ 8 k tokens) | NV‑FP4 (4‑bit) | FP8 (or FP16) | Weight size dominates; FP8 KV brings little extra benefit and may add unnecessary code‑path complexity. |
| Long‑context RAG / agent (≥ 64 k tokens) | NV‑FP4 (4‑bit) | FP8 (2× KV memory reduction) | KV‑cache becomes the primary VRAM consumer; FP8 preserves attention quality while freeing memory for longer contexts. |
| Edge or heavily memory‑constrained deployments | NV‑FP4 (4‑bit) | TurboQuant 4‑bit‑nc (if > 2× KV reduction is required) | TurboQuant’s larger memory win can be worth the modest accuracy/throughput penalty, but it must be validated on the target workload. |
--kv-cache-dtype fp8 in vLLM or the equivalent ModelOpt/Transformer‑Engine path for NV‑FP4 on Blackwell [FP8 KV‑cache].By separating weight and KV‑cache precision decisions, infrastructure teams can exploit the high‑ROI combination of 4‑bit weights plus an FP8 KV‑cache for most production workloads, reserving TurboQuant only for extreme memory‑pressure use‑cases.
Quantization error grows as the number of parameters shrinks because each weight carries more information. NVIDIA’s own measurements show that large models (≥ 70 B) retain about 99 % of full‑precision accuracy after conversion to NVFP4, while 7 B–14 B models only keep 95–98 % 【1†content】. The same trend appears with FP8: on six 70 B‑class models the average quality loss on a suite of reasoning benchmarks (MMLU‑Pro, HumanEval+, GPQA, MATH‑500) is ≈ 0.4 points, which is within run‑to‑run noise, whereas aggressive 4‑bit schemes lose 1.6 points (AWQ‑4) and 1.9 points (GPTQ‑4)【2†content】.
The dynamic range of FP8 also protects small models from the “cliff” that can appear with 4‑bit formats. In narrow‑tail domains such as numerical reasoning and long‑context tasks, FP4 introduces 1–3 % deltas that compound, while FP8’s error remains flat until the model size drops below roughly 14 B.
Throughput gains follow the same pattern. FP8’s native tensor‑core support doubles the theoretical FLOP rate versus BF16, but real inference is limited by either compute or memory bandwidth. For a 32 B model that is bandwidth‑bound, FP8 delivers a ~2.2× speedup, whereas an 8 B model that is compute‑bound sees only ~11 % improvement because the GPU cannot fully exploit the extra bandwidth 【3†content】. This asymmetry mirrors the observation that FP8 lifts decode throughput from 1.0× (BF16) to ~1.8× for large models, but the gain collapses for smaller, batch‑size‑limited workloads 【4†content】.
Given these facts, the safest choice for serving dense models under ≈ 14 B parameters on Hopper or Blackwell GPUs is FP8—it preserves accuracy with near‑zero regression and provides a modest but predictable throughput boost. 4‑bit NVFP4 should be reserved for larger dense or MoE models where the marginal quality loss (1–3 %) is drowned out by benchmark noise and the extra throughput (up to 3×) becomes worthwhile.
Post‑training quantization (PTQ) converts a finished model to NVFP4 in a single pass. It uses a small calibration set (≈100–500 unlabeled examples) and runs in minutes on a single GPU. Modern PTQ algorithms such as MR‑GPTQ achieve 96 %–99 % accuracy recovery versus FP16 on a range of benchmarks【https://www.emergentmind.com/topics/nvfp4-quantization-algorithm】. PTQ is fast and requires no access to the original training pipeline, but it cannot recover the small accuracy gaps that arise from quantization noise in sensitive layers【https://atomic.chat/blog/guides/what-is-nvfp4】.
Quantization‑aware training (QAT) inserts fake‑quantize nodes into the training graph and uses a straight‑through estimator so the weights learn to reside on the 4‑bit grid. This incurs full training cost, but the model can regain accuracy that PTQ cannot. A concrete production example is Kimi’s QAT‑produced K2.7‑Code NVFP4 checkpoint, which matches the publisher’s 4‑bit baseline (SWE‑Bench Verified 74.3 vs 74.1, Terminal Bench 2.1 72.5 vs 71.9)【https://blog.umans.ai/blog/glm-5-2-nvfp4-not-worth-serving】.
A contrasting case is the PTQ conversion of GLM‑5.2’s FP8 release. The community‑run experiment reached 200 + tokens / s with a median time‑to‑first‑token under 2 seconds, but the resulting quality was judged insufficient for coding workloads and the trial was terminated【https://blog.umans.ai/blog/glm-5-2-nvfp4-not-worth-serving】. This illustrates that raw throughput gains do not guarantee acceptable downstream performance.
Most published NVFP4 checkpoints are mostly‑4‑bit:
| Component | Typical Precision in a “4‑bit” checkpoint |
|---|---|
| MoE expert linear layers | NVFP4 |
| Shared expert, attention, embeddings, routers | BF16 / FP8 (higher precision) |
| KV cache | FP8 (or higher) |
| Non‑linearities, layer‑norm, output head | BF16 / FP32 |
NVIDIA’s Nemotron 3 Ultra checkpoint explicitly leaves embeddings, attention, and the KV cache in higher precision while quantizing only the MoE experts【https://developer.nvidia.com/blog/creating-the-nvidia-nemotron-3-ultra-nvfp4-checkpoint-with-nvidia-model-optimizer】. The same fine‑print appears in the GLM‑5.2 NVFP4 card description【https://blog.umans.ai/blog/glm-5-2-nvfp4-not-worth-serving】.
By aligning the checkpoint provenance with the workload’s quality tolerances and the hardware generation (Hopper FP8 vs. Blackwell NVFP4), teams can maximize both throughput and reliability in production LLM serving.
Fit‑first rule – 70 B models in FP8
An FP8‑quantised 70‑billion‑parameter model consumes roughly 70 GB of VRAM, which comfortably fits an 80 GB H100. This makes the H100 the default, cost‑effective GPU for modest‑scale deployments where the KV‑cache does not dominate memory usage【source: https://lyceum.technology/magazine/how-much-vram-for-70b-model/】.
When KV‑cache or long contexts bite
If the serving workload requires large contexts (e.g., > 128 K tokens) or high per‑GPU batch sizes, the H200 becomes the next step. It retains Hopper’s compute pipeline but expands memory to 141 GB HBM3e and bandwidth to 4.8 TB/s, delivering up to 1.9× higher inference throughput for Llama 2 70 B in FP8 compared with the H100【source: https://www.gmicloud.ai/en/blog/best-gpu-hardware-for-ai-inference-full-comparison】. The extra headroom also enables single‑GPU topologies without tensor‑parallel splits.
High‑traffic, cost‑per‑token calculus
At scale, the B200 shines only when its 4‑bit “NVFP4” path is used. Measured FP4 throughput on a B200 yields ~42 % lower cost per token than an H100 running FP8, while B200 FP8 is ~19 % more expensive per token than the H100 at on‑demand rates【source: https://www.spheron.network/blog/nvidia-b200-complete-guide/】. The cost advantage of FP4 depends on sustained utilisation; idle capacity erodes the per‑token saving because the B200’s hourly rate is higher than the H100’s.
When you need > 70 B or > 128 K context
The B200’s 192 GB HBM3e memory pool (the largest among Hopper/Blackwell GPUs) accommodates models exceeding 100 B parameters or contexts that demand 30–50 GB of KV cache. This eliminates the need for multi‑GPU tensor‑parallel splits and keeps latency low for extremely long prompts【source: https://www.spheron.network/blog/nvidia-b200-complete-guide/】.
Cross‑format economics – the Kimi case
Real‑world serving of the Kimi K2.5/K2.6 checkpoint on a B200 using NVFP4 costs 2.71–2.95× less per million tokens than an H200 running INT4 across the 30–90 tok/s / user band (e.g., $0.140 M vs $0.413 M at 32 tok/s / user)【source: https://inferencex.semianalysis.com/blog/b200-nvfp4-vs-h200-int4-kimi-k2-vllm-perf-per-dollar】. The benefit grows with higher interactivity because NVFP4’s higher throughput pushes the B200 into operating regimes unavailable to Hopper.
Mixed‑fleet checkpoint portability
NVIDIA’s NVFP4 checkpoints are single‑file assets that can be served on both Blackwell and Hopper. On Blackwell the model runs W4A4 (4‑bit weights + 4‑bit activations), while on Hopper the same checkpoint automatically falls back to W4A16 (4‑bit weights + 16‑bit activations) via framework conversion【source: https://developer.nvidia.com/blog/creating-the-nvidia-nemotron-3-ultra-nvfp4-checkpoint-with-nvidia-model-optimizer/】. This alleviates operational friction for fleets that span generations.
Decision flow – start with an H100 if a 70 B FP8 model fits comfortably; move to H200 when VRAM‑bound contexts dominate; upgrade to B200 when token‑cost efficiency (FP4) or > 100 B model size is required, ensuring you keep utilisation high enough to amortise the higher hourly price. All price figures reflect a snapshot in 2026 and should be refreshed for each procurement cycle.
The same model on the same GPU can prefer a different numeric format depending on how it is used. Two serving phases dominate LLM traffic:
Consequently, quantization delivers its biggest gains on decode‑heavy, interactive workloads. When the request mix contains many short prompts that quickly transition to decode, the reduced data volume of FP4 (or NVFP4 on Blackwell) translates directly into higher tokens‑per‑second. On the other hand, compute‑bound prefill workloads see only modest speedups, because the tensor cores already process the data at near‑peak FLOP rates and the saved bandwidth is not the limiting factor.
These trends have been observed on Hopper H100/H200 GPUs, where native FP8 runs efficiently, and on Blackwell B200/GB200 GPUs, where NVFP4 tensor cores provide roughly 2× the throughput of FP8 on the same silicon — and FP8 already delivers about 2× versus BF16 【source1】. The combined effect gives FP4 on Blackwell a theoretical 4× gain over BF16, though real‑world decode throughput typically falls in the 1.5–2× range because activations, the KV cache, and kernel overhead remain in higher precision 【source5】.
In burst‑load scenarios, queuing can inflate TTFT dramatically. Deployers sometimes accept a lossier 4‑bit KV cache (e.g., W4A4) to keep latency low, even though it reduces absolute throughput, because the overall user‑experience metric (TTFT) outweighs raw token‑per‑second efficiency.
By aligning the numeric format with the dominant workload characteristics—prefill compute intensity, decode bandwidth pressure, batch size, and context length—infrastructure teams can extract the maximum ROI from Hopper’s FP8 or Blackwell’s NVFP4 capabilities while staying within acceptable quality limits.
Choosing the lowest‑precision format is an empirical decision that should be gated by measurable quality rather than by hardware defaults. The following validation workflow translates the abstract “FP8 vs. FP4” trade‑off into concrete, repeatable steps.
VLLM_USE_FLASHINFER_MOE_FP4=1 flag SVP4 vs MXFP4 guide, “vLLM-specific details”. TensorRT‑LLM gained full NVFP4 support in version 0.17; earlier releases may fall back to slower code paths TensorRT‑LLM, “FP4 support”. Re‑run the entire gate whenever the stack (ModelOpt, vLLM, TensorRT‑LLM) is upgraded, and store the resulting precision policy alongside the model/GPU pair metadata.By codifying this gate, infrastructure teams can systematically exploit the extra throughput of Hopper’s native FP8 and Blackwell’s NVFP4 while guaranteeing that the chosen precision stays within the application’s quality envelope.