An AI capacity plan should begin with the work the fleet must complete. Training a model, processing a long prompt, and generating an interactive response create different demands on accelerators, memory, networking, and operations. Those differences can change the best investment even when the headline accelerator count stays the same.
They do not make existing GPU fleets obsolete. They make workload evidence essential to procurement. The executive decision is whether to improve a shared fleet, separate serving phases in software, add specialized capacity, or combine those approaches—and how to preserve service quality while doing so.
A changing mix is a planning input
Deloitte's 2026 technology predictions, published in November 2025, forecast that inference would account for roughly two-thirds of AI compute in 2026. Deloitte also expected substantial continued demand for advanced chips and large data centers. The two-thirds figure is a forecast, not a measured census of hyperscaler fleets, and it does not establish the mix at an individual operator.
For a capacity plan, measure that mix directly. Separate model development, fine-tuning, batch inference, interactive inference, and agentic tasks. Record the demand distribution and service requirements for each. A fleet dominated by document ingestion may need a different balance from one serving short prompts with lengthy responses, even when both are called inference infrastructure.
The original purchase rationale is relevant history. The next investment should be justified against the workloads expected over its useful life.
Training, prefill, and decode require separate measurements
Training updates model parameters. Its bottlenecks can include arithmetic throughput, memory capacity, data delivery, synchronization, and recovery. Which one dominates depends on the model, parallelism strategy, cluster scale, and implementation.
For autoregressive language-model serving, prefill processes the input sequence and constructs state used during generation; decode produces subsequent tokens. Prefill commonly places heavy demands on compute. Decode can be constrained by memory movement, especially at lower batch sizes. These are useful starting hypotheses, not fixed laws for every model and serving configuration.
The DistServe research demonstrates why phase-level measurements matter. It separates prefill and decode onto different GPUs and jointly considers allocation, parallelism, and network bandwidth. Its service objective includes both time to first token and time per output token. The evidence comes from the authors' evaluated systems and workloads; it is not a universal saving for any fleet.
For procurement, translate those distinctions into a small measurement set:
| Workload | Measure before selecting infrastructure |
|---|---|
| Training and fine-tuning | Time to the required evaluation target, memory fit, communication overhead, queueing, and failure/recovery cost |
| Interactive prefill | Input-length distribution, time to first token, concurrency, cache reuse, and transfer overhead |
| Interactive decode | Output-length distribution, inter-token latency, memory pressure, batch size, and completed requests meeting the service objective |
| Batch inference | Accepted output per unit of cost and energy, completion deadline, and tolerance for interruption |
| Agentic tasks | End-to-end accepted-task rate, including tool calls, retries, host work, and review requirements |
A token-count ratio alone does not determine the right architecture. It must be interpreted alongside latency, model behavior, batching, and the measured bottleneck.
What specialization research establishes
The October 2025 SPAD paper models specialized prefill and decode chips. End-to-end simulations using production traces report 19–41 percent lower hardware cost and 2–17 percent lower thermal design power than modeled baseline clusters at the same performance.
These are simulation results for specified comparisons. Hardware cost is not total cost of ownership, and thermal design power is not measured facility electricity consumption. The range is neither a universal ceiling nor a promised saving. It is evidence that phase-specific hardware is worth evaluating under the conditions studied.
A buyer needs an additional comparison: the proposed architecture against a tuned version of the fleet they can actually operate. Include batching, caching, quantization where acceptable, scheduling, and the relevant serving software. Do not count benefits that the baseline already captures, or assume a research configuration represents an available, supported production product.
The operating cost of separation
Separate pools create separate responsibilities. The organization must manage capacity balance, admission control, cache transfer, observability, failure recovery, and software compatibility across the handoff. Those costs belong in the investment case.
If demand changes rapidly, one pool may become constrained while another has spare capacity. If a network handoff dominates latency, faster phase execution may not improve the user experience. If an unfamiliar platform increases incident frequency or recovery time, an apparent hardware saving can disappear in the service economics.
A shared fleet can therefore be the better choice. It can absorb changes among training, fine-tuning, and serving, and it may fit the team's operational capabilities. Specialization becomes attractive when its measured advantage is large enough, persistent enough, and operationally achievable enough to outweigh those benefits.
I would compare three options before authorizing a fleet transition: tune the current pool, separate phases on supported general-purpose hardware, and introduce specialized hardware where a suitable product exists. Evaluate all three against the same demand trace, quality requirements, service objectives, and failure scenarios.
Power and network belong in the same decision
The IEA's 2025 Energy and AI report projects global data-center electricity use at around 945 TWh in 2030 in its Base Case and identifies grid constraints as a potential source of project delays. That outlook supports early coordination with utilities and facilities teams; it does not establish that power is the binding constraint at every site.
For each candidate deployment, distinguish reserved power, commissioned power, equipment nameplate ratings, and metered consumption. Measure energy per accepted task across the service boundary. Include cooling and other facility overhead where the investment decision spans the facility.
Treat the network handoff with the same discipline. In a disaggregated serving design, prefill state has to be available to the decode workers. Budget its transfer, contention, and recovery behavior within the end-to-end latency target. A component benchmark that excludes this handoff cannot settle a system-level procurement decision.
A licensing agreement is not a workload benchmark
Groq's December 2025 announcement described a non-exclusive licensing agreement with NVIDIA for inference technology, with some personnel joining NVIDIA and Groq remaining independent. Describing it as NVIDIA acquiring Groq is inaccurate.
The announcement is relevant context for inference specialization. It cannot establish the availability, price, performance, or operating economics of a particular deployment. Those require product-specific evidence and a comparable test.
The capital review should expose the break-even point
The investment memorandum should show where each option wins and where it stops winning. I would require:
- A demand baseline: observed workload distributions, service objectives, contracted demand, and scenarios for growth or change.
- A measured alternative: comparable accepted-task throughput, latency, energy, and failure behavior on configurations the team can support.
- A complete cost boundary: equipment, network, facilities, software, migration, staffing, support, financing, and residual value.
- A flexibility assessment: the effect of a different model, context length, traffic pattern, or training requirement on each option.
- A staged commitment: a pilot with explicit acceptance criteria, a rollback path, and evidence required before expanding the purchase.
The useful question for the CFO is whether the additional commitment improves the cost and reliability of delivering the required service. It is not whether a phase-specific benchmark makes an existing asset look unfashionable.
A fleet becomes economically stranded when its remaining service value cannot justify the costs and constraints it carries. A shift toward inference alone does not prove that condition. Workload-matched planning gives leaders a way to distinguish assets that need tuning, assets that should be redeployed, and investments that merit a different architecture.
Originally published 6 May 2026. Revised 13 September 2026 to distinguish forecasts, simulations, and operating evidence; correct the NVIDIA–Groq reference; and replace categorical stranded-capital claims with conditional investment criteria.