Fine-tuning needs more memory than inference
During training, gradients and optimizer states sit in memory next to the weights. A common rule of thumb for full fine-tuning with an Adam-style optimizer in mixed precision is roughly 16 bytes per parameter. Activation memory, which depends on batch size and sequence length, comes on top.
By that estimate, full fine-tuning of a 7B model does not fit on one 80GB GPU and needs a method that shards states across GPUs, such as ZeRO or FSDP.
LoRA and QLoRA
LoRA freezes the original weights and trains only small adapter weights. Gradients and optimizer states exist only for the adapters, so memory drops sharply. QLoRA also quantizes the frozen weights to 4-bit to save more memory.
These methods need far less VRAM than full fine-tuning, so 24GB or 40GB GPUs can handle mid-size models. Result quality depends on the task and data, so check it against an evaluation set.
Deciding GPU type and count
- Pick the method first: full fine-tuning, LoRA or QLoRA.
- Run a short training job to see whether one GPU runs out of memory and how long each step takes.
- If you need several GPUs, compare the per GPU-hour price of 8-GPU instances with 1-GPU instances. The interconnect (SXM vs PCIe) can change multi-GPU training speed.
- Total cost = price per GPU-hour × number of GPUs × training hours.
Controlling cost during training
Training jobs are often paused and restarted. Storage for checkpoints and idle time with the instance still running both count toward cost. Shut the instance down as soon as training ends.
How to Check on Perhour
Compare high-throughput training instances on the A100 price and H100 price pages, or explore the H100 vs A100 comparison to evaluate training cost efficiency.