How much GPU memory does your model need?
Memory, not compute, is what stops most models running. It is also the single specification that most often decides which GPU you should buy, and why.
What decides this requirement.
Weights first, then concurrency.
At inference, model weights must sit in GPU memory. In half precision that is roughly two bytes per parameter, so a 70B model needs around 140GB before anything else. Quantisation to eight or four bits cuts that substantially, at a quality cost that is workload-dependent and worth measuring rather than assuming.
Whatever remains after weights goes to the KV cache, which grows with concurrent requests and context length. This is why concurrency targets drive memory sizing as much as model size does, and why teams serving long contexts run out of memory sooner than they expect.
The practical answer is usually less card than teams assume for small models, and more than they assume for long-context serving.
Three or four times the weights, before activations.
Training holds weights, gradients of matching size, and optimiser state that for Adam-family optimisers typically doubles the parameter memory again. Activations add more, scaling with batch size and sequence length.
A useful rule of thumb: budget several times the raw parameter memory before you have accounted for activations at all. That is why training is inherently multi-device for anything but small models, and why sharding techniques exist.
Mixed precision and gradient checkpointing both change the arithmetic meaningfully, trading compute for memory. Which trade is right depends on whether you are memory-bound or compute-bound, which is worth establishing before buying.
Memory is an economic specification.
Because memory decides how many devices hold your model, a larger-memory part reduces node count rather than merely running faster. That is the genuine argument for H200 over H100, or B300 over B200: not benchmark speed, but avoiding the purchase of extra nodes to solve a memory problem.
Buying more smaller-memory nodes to work around a memory ceiling is the expensive way to solve it, and it is a common pattern. We size against the actual model and workload rather than a generic recommendation.
The per-SKU memory specifications are on the GPU catalogue, and the comparison logic in H100 versus H200.
Start from what is verified.
Current verified lines with quantities, lead times and indicative pricing are public on the live inventory. Anything not listed becomes a sourcing requirement with a first pass inside one business day.
Asked first, answered straight.
Roughly 140GB for weights alone in half precision, before KV cache for inference or gradients and optimiser state for training. Quantisation reduces it substantially at a quality cost worth measuring.
It can, and how much is workload-dependent. Eight-bit is frequently near-lossless for many tasks; more aggressive quantisation needs evaluating against your own benchmarks rather than published ones.
Because it holds gradients and optimiser state alongside the weights, typically several times the parameter memory before activations, which scale with batch size and sequence length.
If memory is your binding constraint, almost certainly, because it reduces node count rather than just running faster. If it is not, you are paying for headroom you will not use.
Working through this on a real requirement?
Twenty minutes with the desk, no pitch and no quote at the end of it. Tell us roughly what you need and we will come back within one business day.
Sent. We are on it.
Your enquiry has landed with the desk. Acknowledged within one hour.