Start with VRAM
Most of the GPU memory used for inference goes to the model weights. You can estimate their size with:
Weight memory ≈ number of parameters × bytes per parameter
- FP16 or BF16: 2 bytes
- 8-bit quantization: 1 byte
- 4-bit quantization: about 0.5 bytes
For example, a 7-billion-parameter (7B) model needs about 14GB in FP16 and about 3.5GB in 4-bit.
Memory beyond the weights
Real usage is higher than the weights alone. The KV cache for text being generated, activations and framework overhead are added on top. The KV cache grows with context length and with the number of concurrent requests. With long contexts or many concurrent requests, it can need as much memory as the weights or more.
Pick a GPU with headroom above the weight estimate. For an exact figure, run the model in your inference engine with your real context length and measure.
Candidates by VRAM
- 24GB (L4, A10G): small models or quantized mid-size models with few concurrent requests
- 40GB or 80GB (A100): larger models, longer contexts, more concurrent requests
- 80GB (H100): when you need higher throughput than an 80GB A100 at the same memory size
A model that does not fit on one GPU can be split across several, but cross-GPU communication changes the setup and the cost.
Comparing cost
Inference services often run around the clock, so hourly price gaps add up directly in the monthly bill. Among GPUs that meet your memory needs, compare providers starting from the lowest price per GPU-hour, then measure throughput under the same load and decide by cost per request. Use the VRAM filter on the Perhour price table to compare by memory size.
How to Check on Perhour
Explore the L4 price, A100 price, and H100 price pages to compare hourly rates across providers, or filter by memory size on the 24GB VRAM and 80GB VRAM pricing pages.