An ordinary computer spends much of its time waiting for something to do. An AI chip is built to do millions of simple calculations at the same time, and when it is training or answering requests it runs close to full speed for hours. On top of that, AI models are very large, so the chip is constantly fetching numbers from memory, and moving data uses far more energy than calculating with it. Many such chips packed into one rack is why AI racks need many times the power of ordinary ones.
Many calculations at once
AI models work by multiplying large tables of numbers. Chips designed for this, often called graphics processing units or accelerators, contain thousands of small calculating units working side by side. Researchers at MIT, in a survey of the field, note that highly-parallel compute paradigms are very commonly used for this work. More units working at once means more power drawn at once.
Moving data costs more than calculating
The same MIT survey finds that for these models, the bottleneck for processing is in the memory access, and that DRAM accesses require up to several orders of magnitude higher energy than computation. In other words, fetching a number from the main memory can cost hundreds of times more energy than doing arithmetic with it. Much of the design of AI chips is about keeping data close to where it is used.
From one chip to a rack
| Level | Figure | Source |
|---|---|---|
| One NVIDIA H100 SXM chip | Up to 700 W, configurable | NVIDIA H100 specifications |
| One NVIDIA DGX B200 system, eight GPUs | About 14.3 kW maximum | NVIDIA DGX B200 specifications |
| A rack of AI GPUs | Often 40 to 100 kW | PNNL, ASHRAE and NEMA framework |
| A traditional rack of ordinary servers | 5 to 10 kW | PNNL, ASHRAE and NEMA framework |
The newest rack-scale systems, such as NVIDIA's GB200 NVL72 with 72 GPUs in one rack, are designed around liquid cooling because of the heat this power becomes.
What is not certain
- Chip power figures are maximum ratings from the manufacturer; actual use depends on the work and the settings.
- Rack densities vary by design and are rising; the ranges here are guides, not measurements of any one site.
- New chips do more work per unit of energy, so power per chip and energy per task can move in different directions.
Sources and method
- Sze, Chen, Yang and Emer, Efficient Processing of Deep Neural Networks: A Tutorial and Survey, arXiv:1703.09039, version of 13 August 2017. https://arxiv.org/pdf/1703.09039
- NVIDIA, H100 product page and specifications. Read 29 October 2026. https://www.nvidia.com/en-us/data-center/h100/
- NVIDIA, DGX B200 product page and specifications. Read 29 October 2026. https://www.nvidia.com/en-us/data-center/dgx-b200/
- NVIDIA, GB200 NVL72 product page, updated 27 August 2026. https://www.nvidia.com/en-us/data-center/gb200-nvl72/
- PNNL, ASHRAE and NEMA, AI Data Center Energy Performance Framework: Energy and Thermal Efficiency. Undated web page, read 29 October 2026. https://www.ashrae.org/technical-resources/topics-and-initiatives/ai-data-center/energy-and-thermal-efficiency
Method: each link was opened on 29 October 2026 and every figure was taken from the page cited beside it. No figure was estimated.
Read next
This article provides general information, not investment advice or an offer of securities.