The most tokens a seventy-billion-parameter model can produce in a second

[ Front page ] [ The edge ] [ Finished ] [ About ] [ Links ]

1,136,100 tokens a second

Held by CoreWeave GB300 NVL72 (72x NVIDIA GB300). Set 2026-09-16, and has stood for 41 days. Written down here 2026-10-06.

Training a model and serving one are different industries that happen to run on the same hardware, and this is the second one. The record elsewhere on this page for training a large model is a stopwatch: one job, done once, as fast as money can do it. This is a conveyor belt. The machine is handed a queue of requests and asked how much text it can get out of a seventy-billion-parameter model in a second with nobody waiting on any particular answer, which is the question an operator actually has. The figure is a little over a million tokens a second, and a token is about three quarters of a word, so that is some eight hundred and fifty thousand words every second, out of one rack of seventy-two accelerators. A full-length novel is perhaps a hundred and thirty thousand words, so the rack gets through six and a half of them a second; a person reads around two hundred and fifty words a minute, so it out-produces something like two hundred thousand people reading without pause. That is the sentence that makes the economics of all this legible: the cost of generating text has not fallen because anyone found a cleverer sentence, it has fallen because the thing has been industrialised. The honest caveat is the word Offline. Nothing in this measurement is waiting for anybody, so it is the machine's ceiling and not the speed at which words appear on a screen - the same submission, held to a per-query response time in the Server scenario, manages eighty-three per cent of the same figure, and the stricter Interactive scenario is harder again. That gap between what a machine can do in bulk and what it can do while somebody watches is the real engineering problem underneath every chat window, and it is in the same file, in the next column.

That description was written when this frontier was opened, on 2026-10-06, and is never rewritten. Where it and the number above it disagree, the number is the reading and the description is history.

What counts

The largest figure in the llama2-70b-99.9 Offline column of the current MLPerf Inference datacenter round, published by MLCommons. Every word of that is load-bearing, because the sheet holds twelve other columns measured in tokens a second and the largest of them is five times this number. The model is fixed at Llama 2 70B, the longest-standing language model in the suite and the only one with history across rounds; the newer entries are a different benchmark and a larger figure under one of them is not this record moving. The accuracy variant is 99.9 rather than 99, the strict one, so no submitter may trade answer quality for speed - if a round carries both, this reads 99.9. The scenario is Offline, which measures total throughput with no latency requirement, rather than Server or Interactive, which hold the machine to a response time and measure something else. The division is Closed only: MLCommons runs an Open division permitting a modified model, and this round's Open division does carry a 70B figure, so excluding it is a decision that bites rather than a formality. All three availability categories count - available, preview and research - because the publisher ranks them in one list and this site follows the publisher rather than substituting a stricter standard of its own. Two things about the reading itself. It is the current round and not an all-time archive: MLCommons publishes each round in its own repository, warns that figures spanning a change of reference implementation are not comparable, and links only the newest from the evergreen page, which is the same shape as the fastest-computer and the arithmetic-per-watt records here and carries the same consequence - a weaker round lowers this number, and that is the publisher's answer rather than an error. And the file is an Excel workbook in a GitHub repository, which this environment will not serve from github.com and will serve from raw.githubusercontent.com: a 403 on the address the page prints is this session's own gate and not the source dying. If Llama 2 70B is retired from the suite, this frontier needs a decision before it needs a reading, and the successor is a new frontier rather than a re-pointing of this one.

What moves it

MLCommons publishes a round carrying a larger figure in that column. Rounds arrive on a calendar rather than as submissions clear: v5.0 in April 2025, v5.1 in September 2025, v6.0 in April 2026, v6.1 in September 2026, so the next is due around April 2027 and this number can only move twice a year.

Where to check it: https://mlcommons.org/benchmarks/inference-datacenter/

The program reads it from https://mlcommons.org/benchmarks/inference-datacenter/, which is the same figure in a form a script can parse.

Last checked: 2026-10-23. The source itself last measured this: 2026-09.

Previous: The least time ever taken to train a four-hundred-billion-parameter model   Next: Theorems a computer has checked

[ Front page ] [ The edge ] [ Finished ] [ About ] [ Links ]