The Distributed Training Bottleneck: Interconnect Bandwidth
When scaling LLM training across 4, 8, or dozens of GPUs, aggregate compute is rarely the primary constraint. Instead, inter-GPU communication bandwidth dictates training velocity.
During gradient synchronization (All-Reduce) and tensor communication passes, inadequate interconnect bandwidth can cause GPU compute utilization to collapse below 30%.
SXM vs PCIe Form Factors
Enterprise GPUs like NVIDIA A100 and H100 ship in two primary physical form factors:
-
SXM (Integrated Socket Architecture):
- GPUs mount directly to an NVLink baseboard.
- H100 SXM5 delivers 900 GB/s bidirectional NVLink bandwidth per GPU.
- An 8-GPU node functions as a unified full-mesh topology, providing non-blocking all-to-all communication necessary for large-scale model parallelism.
- Tradeoff: Premium hourly rental rates (typically 15-30% higher than PCIe).
-
PCIe (Standard Expansion Card Architecture):
- GPUs install into standard server PCIe Gen4 or Gen5 slots.
- Slot bandwidth tops out at 64 to 128 GB/s bidirectional. Multi-GPU NVLink bridges are restricted to 2-way pairings or absent entirely.
- Tradeoff: Substantially lower hourly costs, but creates communication latency during Tensor Parallelism and DeepSpeed ZeRO-3 operations.
Deployment Guidelines
- Choose SXM: Pre-training and full fine-tuning of 13B+ parameter models, Megatron-LM tensor parallel runs, and high-frequency parameter synchronization.
- Choose PCIe: Standard Distributed Data Parallel (DDP) vision training, independent batch inference, and single-GPU parameter-efficient LoRA runs.
How to Check on Perhour
Filter instances by form factor on the H100 price and A100 price pages to select the optimal architecture for your training budget.