Inference Gains Hit 2.7x to 5.7x in Record MLPerf Round

The best per-accelerator server result on MLPerf’s DeepSeek R1 reasoning test improved 5.7x in a year, according to MLPerf Inference v6.1 results released Wednesday by MLCommons, the benchmark setter for the AI industry.

The round set participation records, with 30 organizations submitting 486 data center and edge performance results. Five organizations and one individual contributor submitted for the first time. MLCommons listed five new processors, accelerators and platforms in the results, among them NVIDIA’s Rubin GPU and the Vera Rubin NVL72 rack-scale platform in preview. MLCommons added two tests, one for end-to-end retrieval-augmented generation (RAG) and one for agentic inference at the edge.

Round-Over-Round Gains

The DeepSeek R1 result sits at the top of a 2.7x to 5.7x range the consortium reported for per-accelerator gains over the past year. The best per-accelerator server result on the visual language model test improved 2.99x against v6.0 six months ago. Llama 2 70B, in its sixth round, showed a median per-accelerator gain of 5.58x since v4.0.

MLCommons attributed the range to software and new platforms, and it named next-generation accelerators and lower-precision formats as the two key drivers behind the Llama 2 70B gain. Miro Hodak, MLPerf Inference working group co-chair, said in a press briefing that early rounds ran FP8, while almost all v6.1 results used FP4.

Submitters also shifted which models they ran. GPT-OSS-120B drew 112 submissions, the most of any workload in the suite, unseating Llama 2 70B from the top spot. Version 6.1 also opens speculative decoding to GPT-OSS in the interactive scenario, where it joins DeepSeek R1.

The benchmark treats power measurement as optional, and many submissions skip it, said David Kanter, founder of MLCommons and head of MLPerf, when asked during the briefing Q&A whether submitters recorded wall power during runs. Shruti Koparkar, an NVIDIA product marketing senior manager, added that her company submitted no measured power this round, and that normalizing against a rack’s rated power falls short “because you are not actually drawing all the power on every workload.”

Systems Behind the Numbers

Systems got bigger and more varied. Multi-node submissions reached 16, up from 13 in v6.0. Crusoe submitted a 512-accelerator run on AMD Instinct MI355X GPUs, the largest system ever entered in the benchmark. Cisco unified NVIDIA H200 and AMD Instinct MI350X GPUs into one inference pool, which Hodak said was the benchmark’s first submission to combine accelerators from the two vendors. Cisco reported 99.9% aggregation efficiency on Llama 2 70B. MangoBoost reported that its LLMBoost software knit 32 GPUs at four sites into a single serving pool across an intercontinental network, achieving 97% scaling efficiency in offline mode.

Hodak tied distributed inference to production practice. “Distributing models across multiple nodes is a trend in the industry, especially if you want to minimize time to first token,” he said. Cloud companies running more than one vendor’s silicon need a way to ship workload between them, he said, and geographic distribution answers a need for resilience.

The new silicon MLCommons listed runs from the desktop to the data center rack. AMD’s Ryzen AI Max+ 395 is a unified-memory APU for local AI, and Intel’s Arc Pro B70 a 32GB workstation GPU for large-model inference. AMD’s Instinct MI350P brings a PCIe option for air-cooled servers. NVIDIA entered its Rubin GPU and Vera Rubin NVL72 platform in preview, the category for hardware due to ship by the next round.

The Benchmark’s New Tests

Both new tests measure multi-step workloads rather than a single pass:

End-to-End RAG measures a full question-answering pipeline rather than a single model call. Four models handle embedding, reranking, document grading and answer generation, and a sufficiency checker sends the query back around the loop up to five times. It reports documents per second for building the vector database and tasks per second for answering, where one question runs a dozen or more model calls.

Edge Agentic Inference replays 20 recorded agent trajectories drawn from real GitHub issues in SWE-bench Verified, 1,007 turns in total, against Qwen3.6-27B at a 32K context window, and reports mean single-stream latency per turn. Accuracy is gated on a separate run against the Berkeley Function-Calling Leaderboard v4.

Ramesh Chukka, who co-chairs the End-to-End RAG task force, said the effort began more than a year ago with member feedback asking for a benchmark that “measures the performance and accuracy from end to end, not just for the inference.”

Atlas Inference, a first-time submitter, ran the edge test on an NVIDIA DGX Spark and on AMD Strix Halo. “For the past few years, serious agentic work meant a datacenter round trip,” the company wrote in its submitter statement. “That assumption is what this submission is meant to retire.” The company reported that it completed the full 1,007 turns on the DGX Spark in under 64 minutes.

Lambda ran the same agentic test on data center silicon, submitting in the edge category, open division, on eight NVIDIA B200 GPUs. Chuan Li, chief scientific officer at Lambda, said the company swapped the benchmark's 27 billion-parameter reference model for Kimi K2.6, a trillion-parameter model from Moonshot AI. Li said Lambda also processed about 9% more output tokens per second on GPT-OSS-120B than it did six months ago, on the same NVIDIA GB300 hardware.

Orrick Industries, another first-time submitter, reported 274,927 tokens per second on Llama 3.1 8B from six NVIDIA B200 GPUs in the open division, where results are not required to be mathematically equivalent to the reference model.

Edge Agentic Inference already runs through MLPerf’s API-centric harness, and the RAG test is set to follow. Sixteen of the 30 submitting organizations used the harness this round, up from a single open-division submitter in v6.0. That harness is the foundation of MLPerf Endpoints, a suite that plots each system’s tradeoff between total throughput and speed per user. Endpoints opens rolling submissions in October 2026 and is slated to replace MLPerf Inference for the data center in 2027. The TechArena Take

Version 6.1 brings the suite closer to how AI gets deployed. Since it began in 2019, MLPerf Inference has measured single-shot workloads, which still cover a large share of production work: image recognition, transcription and single-pass answers. RAG pipelines and agentic coding sessions bring two more shapes into the suite. Buyers sizing infrastructure for those workloads now have peer-reviewed numbers to back up vendor assertions.

The range of systems across the round is the other story. One submission ran 512 accelerators. Another ran on a single SoC, with client and server both on the device. Cisco pooled GPUs from two vendors, and MangoBoost’s software held four sites together across an intercontinental network. A benchmark that puts all of that through a robust peer review process serves an industry buying at every one of those scales.

MLCommons is rebuilding its inference benchmark while participation sits at an all-time high. MLPerf Endpoints arrives in 2027, and Kanter said MLCommons is revisiting normalization for future inference benchmarks, calling it “absolutely critical.”