What Is AI Training? Why the Run Needs the Whole Cluster
AI training is the long, planned job that teaches a model on a reserved GPU cluster. The stuck step is the last missing piece: the run does not start until every GPU is live.
EXPLAINED · Mechanism · Last reviewed: 12 Oct 2026 · Next review: after NVIDIA’s Q3 FY2027 results (expected November 2026)
AI training is the long, planned job that teaches a model: thousands of GPUs read a huge pile of data together and write the weights that later answer questions. It is sold as a reserved cluster, not as a single GPU hour. The run does not start until every chip, cable and watt in that cluster is live.
In short. Training is the job that builds the model. Inference is the job that uses it. CoreWeave, a specialist GPU cloud, booked $2.575 billion of revenue in the quarter to 30 June 2026, and 98% of that came from committed contracts, not walk-up hours. Its remaining performance obligations stood at $103.7 billion on that date, of which 41% is due in the first 24 months (about $42.5 billion, our arithmetic). NVIDIA, which is paid first for the chips and the network, reported $89.0 billion of data-centre revenue in the quarter to 26 July 2026. The stuck step is the last missing piece of a complete cluster: a training run will not start at 80% of the GPUs.
Why an investor cares: training is how a lab turns a pile of chips into a model, and how a neocloud turns a hall into contracted cash. The clock that matters is not “when we ordered the GPUs.” It is “when the whole cluster is live.” This page is the training half of the pair; the other half is AI inference.
What is AI training?
Training is the job that teaches the model. The cluster reads tokens of text, images or sound, compares the model’s guess with the right answer, and writes a small change into the weights. Do that enough times, on enough chips at once, and you get a model that can later be used.
The size of that job is now public in two ways: how many chips sit in the cluster, and how much the final run is estimated to cost.
On Meta’s third-quarter 2024 call, Mark Zuckerberg said the company was training Llama 4 “on a cluster that is bigger than 100k H100s.” On 5 April 2025, Meta’s Llama 4 blog said the teacher model, Llama 4 Behemoth, was pre-trained “using FP8 and 32K GPUs” and hit 390 TFLOPs per GPU, on more than 30 trillion tokens. Those are not the same number: 100,000 H100s is the cluster; 32,000 GPUs is Behemoth’s pre-training run.
Epoch AI, which estimates the hardware and energy cost of the final training run, puts GPT-4 at about $40 million (published 15 March 2023) and xAI’s Grok 4 at about $500 million (published 9 July 2025). It says spending on large-scale training has grown about 2.4× a year since 2016. Those are estimates of the run, not the price of the hall.
NVIDIA’s data-centre business is what gets paid when those halls are filled. In the quarter ended 26 July 2026 it reported $89.0 billion of data-centre revenue, 92.5% of its $96.2 billion total (our arithmetic), up 117% from a year earlier. Training sits in the middle of that life cycle: after the data is prepared, before the model is served.
How is a training run different from inference?
Training teaches the model, once, in a planned block. Inference uses it, every time someone asks. The chips can be the same. The way they are sold is not. For the inference meter, see what AI inference is and AI tokens.
| AI training vs AI inference |
|---|
| When it runs: training is a planned job that can last weeks or months before a model launches. Inference runs all day, every time a user or an agent asks. |
| How it is sold: training is usually sold as a reserved block of GPUs on a multi-year contract. Inference is billed as used, per GPU hour or per token. |
| What has to be ready: training needs the whole cluster working as one, including the network between racks. Inference can start on a smaller pool and add copies as requests rise. |
| Who carries idle risk: on a take-or-pay training block, the customer pays for the cluster once it is delivered, busy or not. On per-token inference, the cloud holds spare GPUs ready (our read). |
How it is sold and who carries idle risk are Second Order’s read of CoreWeave’s contracts and of the inference page. Timing and cluster shape: NVIDIA Q2 FY2027 call, 26 August 2026; CoreWeave 10-Q, 30 June 2026.
NVIDIA’s pitch is one platform for both. Its chief executive told analysts on 26 August 2026 that its systems are “great at training, great at inference.” Vera Rubin racks were already running at CoreWeave, Google Cloud, Microsoft Azure, Oracle and Nebius in that quarter.
A benchmark minute is not a production run. In MLPerf Training 6.0, published 16 June 2026, CoreWeave hit the DeepSeek-V3 671B quality target in 2.02 minutes on 8,192 GPUs. Microsoft Azure hit the Llama 3.1 405B target in 7.07 minutes on 8,192 GPUs. Those times measure a compressed test, not the months a frontier lab spends on a full pre-training run. They do show the one fact cash cares about: thousands of GPUs have to work as one.
Why does the run need the whole cluster?
Because training is a single job split across every GPU. Each chip holds part of the model or part of the batch. They have to exchange data on every step. If 20% of the GPUs are dark, the job does not run at 80% speed. It waits.
That is why a neocloud sells clusters, not loose cards. CoreWeave’s 2025 10-K says customers buy “a specified amount of capacity on a take-or-pay basis,” with a weighted-average contract of about five years. How that contract binds is in take-or-pay GPU contracts; the backlog number is in remaining performance obligation.
The network is part of the cluster. NVIDIA’s MLPerf write-up says scale-out training uses InfiniBand or Spectrum-X Ethernet so the racks act as one. Inside a GB200 or GB300 NVL72 rack, NVLink ties 72 GPUs into a single pool. A hall with chips and no fabric is a pile of servers, not a training cluster. Power is the other half of “whole”: a training hall is the dense end of an AI data center. If the last transformer is late, the GPUs sit dark. How busy those chips are once live is GPU utilization.
Who gets paid — and where is the stuck step?
Money leaves the hall before a single trained token is sold. It leaves in a set order.
| Who gets paid on an AI training cluster |
|---|
| 1. The chipmaker, first. NVIDIA is paid when the GPUs and the scale-out switches ship. Data-centre revenue was $89.0 billion in the quarter to 26 July 2026. Vera Rubin racks were already running at CoreWeave and Nebius in that quarter. |
| 2. Memory and packaging, in the same window. A training GPU is not a training GPU until it has HBM stacked on it and a CoWoS package around it. Those steps are paid in Korea and Taiwan before the rack is bolted down. |
| 3. The landlord and the utility, as the hall is fitted. CoreWeave’s cost of revenue in the June quarter was $879 million, 34% of sales. Rent was the largest swing: about $335 million more than a year earlier, with utilities and power up about $87 million (10-Q). |
| 4. The neocloud, once the cluster is delivered. CoreWeave recognised $2.575 billion in the June quarter, 98% from committed contracts. Microsoft was 67% of its 2025 revenue. In September 2025 it signed an OpenAI order of up to about $6.5 billion through 31 May 2031. It does not get paid that backlog until the capacity is live. See what CoreWeave does. |
| 5. The lab, last. The model maker earns after the run finishes and the model is served. Training is a cost. Inference is the bill that can pay it back. |
NVIDIA results, 26 August 2026; CoreWeave 10-K for 2025 and 10-Q to 30 June 2026. Order of pay is Second Order’s read of those filings.
Stuck step: the last missing piece of a complete cluster. That piece can be a GPU, an HBM stack, a CoWoS slot, a switch, or a megawatt. From the neocloud’s chair they look the same: the take-or-pay clock does not start, and the RPO does not turn into revenue, until the customer can run the job.
The backlog makes the wait visible. CoreWeave’s RPO was $60.7 billion at 31 December 2025 and $103.7 billion at 30 June 2026, up $43.0 billion in six months (our arithmetic). Of the June figure, 41% is expected in the 24 months to 30 June 2028, about $42.5 billion (our arithmetic). That is contracted capacity still sitting in the future. A specialist GPU cloud is a neocloud; the larger buyers who also train in-house are hyperscalers.
Our view: a signed training contract is a claim on a future cluster. It is not cash until the last rack is live. Watch live GPU counts, not just RPO.
What would prove this wrong?
- Training is sold by the hour, not the cluster. If CoreWeave’s next results show committed revenue well below 98%, walk-up hours have taken the training seat and the “whole cluster” bind eases.
- Partial clusters train. If labs publish production runs that keep going when a slice of GPUs drop out, without a restart, the all-or-nothing step has moved.
- Inference buys the same reserved blocks. If neoclouds stop splitting training as multi-year take-or-pay and inference as spot or token, the two jobs collapse into one capacity story.
Common misconceptions
- People assume a training run can start on half the GPUs. Actually the job is split across the cluster. If part of it is dark, the run waits.
- People assume the 2-minute MLPerf time is how long a model takes to train. Actually that is time-to-quality on a compressed benchmark, not a full pre-training run.
- People assume the neocloud is paid when the contract is signed. Actually 98% of CoreWeave’s June-quarter revenue was from commitments, but RPO of $103.7 billion is the amount still undelivered. Cash follows a live cluster.
FAQ
What is AI training?
AI training is the long, planned job that teaches a model. A cluster of GPUs reads a huge pile of data together and writes the weights that later answer questions. It is sold as a reserved cluster, not as a single GPU hour.
What is the difference between AI training and inference?
Training teaches the model, once, in a planned block that can last weeks or months. Inference uses the model every time someone asks. Training is usually sold as reserved GPU capacity; inference is billed as it is used, per GPU hour or per token.
How is AI training sold?
Mostly as a reserved block of GPUs on a multi-year take-or-pay contract. CoreWeave said 98% of its revenue in the quarter to 30 June 2026 came from committed contracts, and that those contracts had a weighted-average life of about five years at the end of 2025.
Why does a training run need every GPU?
Because the job is split across the cluster. Each GPU holds part of the model or part of the batch, and they exchange data on every step. If part of the cluster is dark, the run waits. It does not crawl along at 80%.
Who gets paid for AI training?
The chipmaker is paid first, when GPUs and switches ship. Memory and packaging are paid in the same window. The landlord and the utility are paid as the hall is fitted. The neocloud is paid once the cluster is delivered. The lab is paid last, after the model is served.
Is this a list of stocks to buy or sell?
No. Second Order Explained pages explain how the money moves. They do not give buy or sell calls.
Related terms
AI inference · neocloud · take-or-pay GPU contract · remaining performance obligation · GPU utilization · HBM · CoWoS · hyperscaler
Markets and stack doors
Layer: Neoclouds. Also: Hyperscalers, Semiconductors. Markets: US, Korea, Taiwan. Company door: what CoreWeave does.
How we check this
Company figures come from CoreWeave’s 10-K and 10-Q, NVIDIA’s Q2 FY2027 results and call, Meta’s Q3 2024 call and Llama 4 blog, Epoch AI’s training-cost table, and NVIDIA’s MLPerf Training 6.0 post. Each was re-checked on 12 Oct 2026. Where we work something out ourselves, such as a share of revenue or a slice of RPO, it is labelled “our arithmetic”. This page explains how the money moves. It does not give buy or sell calls.
Last reviewed: 12 Oct 2026 · Next review: after NVIDIA’s Q3 FY2027 results (expected November 2026)
Spot an error? Tell us and we will correct it and note the change here.
Sources
- CoreWeave — Form 10-K for the year ended 31 December 2025 — take-or-pay contracts, $60.7 billion of RPO, Microsoft 67% of 2025 revenue, OpenAI order up to about $6.5 billion through 31 May 2031
- CoreWeave — Form 10-Q for the quarter ended 30 June 2026 — $2.575 billion of revenue, 98% from commitments, $103.7 billion of RPO
- NVIDIA — Second quarter fiscal 2027 results (26 August 2026) — $96.2 billion of revenue, $89.0 billion data centre, Vera Rubin at CoreWeave and Nebius
- NVIDIA — Q2 FY2027 earnings call transcript (26 August 2026) — “great at training, great at inference”
- Meta — Q3 2024 earnings call transcript (30 October 2024) — Llama 4 cluster bigger than 100,000 H100s
- Meta — The Llama 4 herd (5 April 2025) — Behemoth pre-trained on 32,000 GPUs at 390 TFLOPs/GPU
- Epoch AI — Training compute costs (table updated 24 November 2025; read 12 October 2026) — GPT-4 about $40 million; Grok 4 about $500 million; 2.4× a year
- NVIDIA — Blackwell sweeps MLPerf Training 6.0 (16 June 2026) — CoreWeave 8,192 GPUs, DeepSeek-V3 in 2.02 minutes
