What Is AI Inference? How Neoclouds Get Paid for It

EXPLAINED · Mechanism · Last reviewed: 10 Oct 2026 · Next review: after CoreWeave’s Q3 2026 results (Nov 2026)

AI inference is running a trained AI model to answer a new request, such as a chat reply, a piece of code or a search summary. Training builds the model once. Inference runs every time someone uses it, and each answer is billed in tokens or GPU hours.

Why an investor cares: inference is the recurring revenue that has to pay back the chips bought for AI. It is how a neocloud turns GPUs into cash, and why clouds now price AI in AI tokens.

What is AI inference?

A model is trained once, on huge amounts of data. After that, every question it answers is inference. The model reads the request (input tokens) and writes the answer (output tokens), and a GPU does the work.

Each answer is cheap. Deloitte notes that asking a model to summarise a document takes only a tiny fraction of the compute needed to train it. But billions of requests add up: Deloitte expects inference to be about two-thirds of all AI compute in 2026, up from a third in 2023 and half in 2025. Source: Deloitte TMT Predictions 2026.

Agents push volume up further. Nebius told shareholders that in agent work, such as coding or search agents, one task drives many model calls, so use grows with how much work is automated, not with how many people log in. Source: Nebius Q2 2026 letter.

AI training vs inference: what is the difference?

Training teaches the model. Inference uses it. The two run on mostly the same chips but are sold and planned differently.

Sources: Deloitte TMT Predictions 2026 (17 Nov 2025); NVIDIA Q2 FY2027 call (26 Aug 2026). The “how it is sold” row is Second Order’s read.

NVIDIA’s pitch is one platform for both: its chief executive told analysts on 26 August 2026 that its systems are “great at training, great at inference”. It also said Groq 3 LPX, its rack-scale system built for fast interactive inference, is in full production. Sources: NVIDIA call and NVIDIA results. For a chip built mainly for Google’s own AI work, see what a TPU is.

How do neoclouds get paid for inference?

There are two meters: the GPU hour and the token.

Sources: Nebius compute pricing and Token Factory catalog, both read 10 Oct 2026. The idle-risk row is Second Order’s read.

Per token, writing costs more than reading: output is 2 to 5 times the input price on the models above. A request that reads 2,000 tokens and writes 500 on Kimi-K3 costs about 1.35 cents (2,000 × $3 ÷ 1 million + 500 × $15 ÷ 1 million = $0.006 + $0.0075, our arithmetic). A million such requests is about $13,500. Sources: Token Factory catalog and Nebius compute pricing.

Per-token selling is new and still small. CoreWeave said booked yearly revenue (ARR) for its managed inference service grew from $1 million to more than $100 million in the few months since launch, and it expects at least $250 million by the end of 2026. Its whole business brought in $2.6 billion in the June quarter, about $10.4 billion a year at that pace, so per-token inference is about 1% of it (booked ARR against quarterly revenue × 4, not like for like; our arithmetic). Sources: CoreWeave call and CoreWeave results.

Nebius does not split out inference revenue. It said work on Token Factory, its per-token service, more than tripled in the June quarter, inside AI cloud revenue of $574.9 million, up 514% on a year earlier. Source: Nebius Q2 2026 letter. The hour-based side is covered in GPU as a service.

Who gets paid when you ask an AI a question?

The money moves in a set order, and the answer is paid for last.

• 1. The chipmaker, first. Paid when the GPU is bought. NVIDIA said sales to neoclouds, industry and enterprise were about $40 billion in its quarter to late July 2026, up 138% on a year earlier, driven by neocloud capacity additions. NVIDIA call.
• 2. The hall and the power, next. Rent and power bills start once racks are switched on. See what an AI data center is.
• 3. The neocloud, per hour or per token. Only when requests flow. CoreWeave raised prices about 25% across its chip types in July 2026, citing demand and the returns customers see as they “shift to inference”. CoreWeave call.
• 4. The model maker or app, last. Paid by the end user, through a subscription or a paid API.

Where is the stuck step?

The spare GPU. To answer fast, a platform needs the model loaded on a GPU before the request arrives. Nebius said on 1 October 2026 that cold starts, the time a model needs to load before serving a single request, leave GPUs idle when demand spikes and new copies spin up, and that this “forces platforms to hold spare capacity just to hit their service-level targets”. It calls this the “idle GPU tax”, and bought a start-up called Inferize to cut it. Terms were not disclosed. Source: Nebius. Our Wire: Nebius acquires Inferize.

Selling spare hours cheaply is the other fix. Since 8 October 2026 Nebius prices its interruptible GPU hours by live demand, from $0.79 an hour for an H100, about 18% of the $4.50 on-demand rate (0.79 ÷ 4.50, our arithmetic). A spare hour sold at the floor earns little; an idle one earns nothing. Sources: Nebius blog and Nebius pricing. Our Wire: Nebius moves spare GPU hours to spot prices.

Behind both sits plain shortage. CoreWeave said growth of its inference service was “constrained only by our near-term capacity”, and NVIDIA expects supply to stay a bottleneck at least to the end of its 2028 financial year, in January 2028. So a neocloud can be short of GPUs overall and still pay for spare ones it holds ready. How busy chips are is the number to watch: see GPU utilization. Our read of what Nebius’s price assumes about it is in the Nebius idle-GPU Analysis.

What would prove this wrong?

• Cold-start fixes work quickly. If Nebius reports clearly higher use of its GPUs after Inferize, spare GPUs stop binding and plain capacity is the only limit.
• Per-token prices fall faster than chip costs. Then the stuck step moves to margin. Watch the Token Factory price list and NVIDIA’s chip prices.
• Per-token selling stalls. If CoreWeave ends 2026 well short of $250 million of managed inference ARR, inference stays mostly a GPU-hour business on long contracts, as in take-or-pay GPU contracts.

What to check next

CoreWeave and Nebius report their September-quarter results in November 2026 (dates not yet set at 10 October); look for the managed inference ARR and Token Factory growth. Nebius shows its live spot price only in its console. NVIDIA guided to about $108 billion of revenue for its quarter to October. The US door, CoreWeave neocloud contracts, tracks the contract side.

Common misconceptions

• People assume inference is cheap, so it needs few chips. Actually each answer is cheap, but the volume is huge: Deloitte expects inference to be about two-thirds of AI compute in 2026.
• People assume inference runs on special cheap chips. Actually most still runs on the same high-end GPUs used for training, though inference-only systems are entering production.
• People assume a neocloud’s inference revenue is all per token. Actually most is still sold by the GPU hour. CoreWeave’s per-token service is about 1% of its revenue pace.

FAQ

Related terms: AI tokens, the unit inference is billed in; GPU utilization; neocloud; GPU as a service.

Stack doors

Layer: Neoclouds. Also: Hyperscalers, AI Data Centers, Semiconductors. Markets: US.

Get the Wire — short daily notes when GPU prices, contracts and capacity move.

Sources