How many GPUs does it take to train a 70B model?
There is no single number, and anyone who gives you one without asking three questions first is guessing. What there is, is a method that gets you to a defensible answer in about twenty minutes.
What decides this requirement.
The model has to fit before it can train.
A 70 billion parameter model in half precision needs roughly 140GB just for weights. Training adds gradients of similar size and optimiser state that, for the common Adam-family optimisers, is typically double the parameter memory again. Before activations, you are already well past what any single GPU holds.
That arithmetic is why 70B training is inherently multi-GPU and usually multi-node, and why memory per GPU changes the node count so sharply. It is also why techniques that shard optimiser state and parameters across devices exist at all.
The practical consequence for a buyer: a card with more memory does not just run faster, it reduces how many nodes you need to buy. That is the real argument for the larger-memory parts, and it is an economic one rather than a benchmark one.
The date sets the cluster, not the budget.
Total training compute is roughly a function of parameter count and token budget. Divide that by the effective throughput your cluster can actually sustain, and you have a duration. Effective is doing a lot of work in that sentence: real clusters do not achieve peak numbers, and the gap between theoretical and realised throughput is largely a fabric and pipeline question.
Most teams do this backwards. They set a budget, buy what it affords, and discover the run takes eight months. The better order is to fix an acceptable completion date, derive the cluster that achieves it, and then decide whether that is affordable, because at least then the trade-off is visible.
If the answer is unaffordable, the honest options are a smaller model, fewer tokens, or a longer timeline. Buying a cluster that cannot finish the run on any useful horizon is the worst of the three.
Utilisation, not specification.
The single largest source of error is assuming near-peak utilisation. Real distributed training loses throughput to communication, pipeline bubbles, checkpointing and failures. A cluster achieving a solid fraction of theoretical peak is doing well, and sizing that assumes otherwise produces a schedule that slips.
Fabric quality is the biggest lever on that fraction, which is why we size the interconnect before the node count. The reasoning is on the LLM training page.
Bring your parameter count, token budget and target date to a twenty-minute call and we will work the arithmetic with you, then price the cluster that actually delivers it.
Start from what is verified.
Current verified lines with quantities, lead times and indicative pricing are public on the live inventory. Anything not listed becomes a sourcing requirement with a first pass inside one business day.
Asked first, answered straight.
Not honestly, without knowing whether you are training from scratch or continuing pre-training, your token budget and your acceptable time-to-train. Those three answers change the cluster by an order of magnitude.
Because it changes how many nodes you need. Weights, gradients and optimiser state for a 70B model far exceed a single device, so more memory per GPU means fewer devices to hold the same model state.
Meaningfully less than theoretical peak. Communication, pipeline bubbles, checkpointing and failures all take a share. Sizing that assumes peak throughput produces a schedule that will slip.
Frequently yes, if it genuinely is a single run. Reserved capacity wins when training is continuous. The comparison is on the buy versus lease page.
Working through this on a real requirement?
Twenty minutes with the desk, no pitch and no quote at the end of it. Tell us roughly what you need and we will come back within one business day.
Sent. We are on it.
Your enquiry has landed with the desk. Acknowledged within one hour.