FP8 and FP4 support in PyTorch: hardware advances and mixed‑precision

FP8 and the newer FP4 formats are emerging as hardware‑level tools to boost AI training speed, especially on Nvidia H100 GPUs that provide dedicated tensor cores for these low‑precision types [2]. By leveraging these datatypes, developers can achieve higher throughput while managing the trade‑off between dynamic range and precision.

Key takeaways

• Nvidia H100 GPUs introduce two FP8 formats: E4M3 (4 exponent, 3 mantissa bits) and E5M2 (5 exponent, 2 mantissa bits) [2].
• FP4 (NVFP4) is added in the Blackwell architecture, complementing FP8 for even lower‑precision workloads [2].
• Mixed‑precision training with FP8 typically uses E4M3 for forward activations and weights, and E5M2 for backward‑pass gradients [2].
• PyTorch can access these formats through the Transformer Engine library, which exposes FP8 APIs for tensor operations [1].
• Real‑world benchmarks on H100‑based instances show step‑time improvements but cost considerations may offset performance gains [1].

Hardware‑level FP8 and FP4 on Nvidia H100

The H100 GPU’s tensor cores support two distinct FP8 formats. The E4M3 format provides a modest dynamic range (up to ±448) with higher precision, making it suitable for forward‑pass activations and weight storage [2]. Conversely, the E5M2 format extends the range to ±57 344, favoring the backward pass where gradients benefit from broader dynamic range despite lower mantissa precision [2]. Blackwell’s addition of NVFP4 and MXFP8 expands the low‑precision toolbox, allowing developers to choose the most appropriate datatype for each stage of training [2].

Integrating FP8 into PyTorch workflows

To exploit these hardware capabilities, developers must use software that exposes the FP8 APIs. The Transformer Engine library provides such support, offering functions that let PyTorch scripts specify FP8 tensors for matrix multiplies and convolutions [1]. While the article does not detail the exact API calls, it demonstrates that modifying a training script to enable FP8 can yield substantial speedups on H100 hardware [1]. The same documentation notes that mixed‑precision training with FP8 still requires careful handling of loss scaling, similar to FP16, because the reduced mantissa bits can cause overflow or underflow if not managed properly [2].

Why it matters

FP8 and FP4 represent a shift toward ultra‑low‑precision training that can dramatically increase FLOPS and reduce memory bandwidth, potentially cutting training time for large models [1]. However, the cost of H100‑based cloud instances, such as AWS’s p5 family, may diminish the economic advantage despite faster step times [1]. As the software stack matures and more frameworks adopt native FP8 support, the balance between performance gains and cost efficiency is likely to improve, making these datatypes a key focus for future AI hardware‑software co‑design.