Why Speed Is the New Moat: Cerebras on the Rise of Fast Inference

A decade ago, Cerebras set out to rethink how computers process AI workloads. Rather than adapt conventional chip architectures, the company designed an entirely new processor and computing system built specifically for AI. That early investment is now paying off as inference becomes an increasingly critical part of the market.

I recently sat down with Cerebras Chief Strategy Officer Andy Hock to discuss the growing demand for its approach amid accelerating deployments and new agreements with OpenAI, AWS, and AMD. As Andy explained, inference speed is becoming a competitive differentiator for businesses building AI-powered applications.

A Better Engine for a New Workload

Cerebras began developing its technology when the scale and direction of the AI market were still uncertain. Its founders recognized that AI-enabled workloads had different computational requirements from graphics, rendering, database management, and other applications traditionally served by general-purpose processors.

“That legacy general purpose compute that was extraordinary to other things was in some sense holding us back as a field from realizing that positive transformational potential,” Andy said.

To address those requirements, Cerebras designed a processor, computing system, and supporting software stack specifically for AI. The result was the Wafer Scale Engine (WSE), a processor built on an entire silicon wafer rather than a conventional individual chip. Andy described it as the world’s largest computer chip, more than 50 times larger than previous designs.

Cerebras now uses the architecture for training and inference, including voice applications, reasoning models, and agentic workflows. According to Andy, its systems can execute certain inference workloads approximately ten times faster than comparable conventional processors.

Why Inference Demand Is Accelerating

The rapid growth of inference reflects a fundamental shift in how organizations use AI, Andy said.

“The reason that we’re seeing a hockey stick of growth in inference is because that’s where AI delivers value back to end users and businesses,” he said.

The launch of ChatGPT in 2022 demonstrated that large language models could perform useful work, driving demand to deploy them in real-world applications. As organizations moved from developing models to using them at scale, inference performance became increasingly critical.

Training remains an important market for Cerebras, but inference places greater emphasis on response times, particularly for applications involving complex reasoning or sequences of agentic actions. The need for speed depends more on the use case than on model size. Cerebras supports both frontier-scale and smaller models, with performance expectations shaped by how users interact with them.

‘Every Millisecond Matters’

Andy drew on his previous experience at Google, where researchers spent years examining how search latency affected user adoption and satisfaction.

“What they found was that every millisecond of latency between when you hit enter or click search and being presented an answer, every millisecond matters,” he said.

The same principle increasingly applies to AI applications. Users accustomed to immediate search results may be less willing to wait for chatbots and agents to generate responses, particularly when those delays accumulate across multistep workflows.

For businesses deploying AI, lower latency can improve responsiveness and enable applications that would otherwise be impractical. The challenge is delivering that performance efficiently as demands on models and workflows increase.

Putting Memory Next to Compute

Andy pointed out that the size of the chip isn’t a gimmick.

“We didn’t make this big chip because it looked cool and we didn’t make it big because it was easy,” he said.

AI workloads, he explained, have three fundamental requirements:

Substantial computing capacity
Memory located close to the processors
High-bandwidth data movement

Conventional architectures place compute and memory on separate chips, connected by relatively narrow links.

“What if we could just put the memory along with the compute and then connect it all at really, really high bandwidth?” he said. “That’s exactly what this is.”

The Wafer Scale Engine uses on-chip SRAM, allowing its cores to access nearby memory with extremely low latency. The architecture delivers several orders of magnitude more memory bandwidth than traditional compute-plus-HBM designs, Andy said.

Building the chip was only the first hurdle. Delivering power, cooling, and data to a wafer-sized processor inside a data center required Cerebras to rethink the surrounding system. For larger models and longer contexts, it distributes workloads across multiple wafers and uses disaggregated inference. Through partnerships with AMD and AWS, complementary hardware can handle prefill (processing the input context), while the Wafer Scale Engine handles decode (generating output tokens).

“If we can produce more fast tokens per wafer scale system by using other hardware, it’s better economics for everybody,” Andy said.

Cerebras also announced a next-generation system at Hot Chips that brings additional memory closer to the wafer, expanding the context it can support. This becomes increasingly important as agentic workloads require longer context windows and greater KV cache capacity.

“These challenges don’t go away,” Andy said. “They just evolve as the market does.”

From National Labs to OpenAI

Cerebras’ customer history reflects the evolution of the AI market: the U.S. Department of Energy, which builds supercomputers for science and energy research, was among its first customers and remains one today. GSK followed, using Cerebras technology for model training and inference to accelerate drug discovery.

The company subsequently formed a strategic partnership with G42 to build a sovereign AI cloud in the UAE and develop open-source Arabic language models.

Most recently, Cerebras announced a deal with OpenAI to deploy 750 megawatts of AI compute, alongside new partnerships with AWS and AMD. “One of our systems today takes about 27 kilowatts and we’re talking about deals... on the order of hundreds of megawatts,” Andy said.

Speed as a Moat

Andy said recent conversations with those in the industry reflected growing demand for improved inference. The questions were practical:

How do I get access?
Which models do you serve?
Can you host a dedicated instance of ours?

He pointed to AlphaSense, which uses AI to deliver financial and market insights to CFOs, traders, and investors. For professionals conducting research, the ability to ask questions and receive immediate answers is “not just a nice-to-have in that business, it’s a need-to-have.”

Andy sees the same competitive implications across industries.

“If you and somebody else are building an AI agent platform for finance or for health or for industrial robotics, and you have fast answers, fast control, fast responses that deliver value sooner and your competitors don’t, that’s a moat.”

The TechArena Take

Cerebras has spent a decade approaching AI compute from first principles, developing an architecture that challenges conventional processor design.

As inference demand accelerates and agentic AI raises expectations for response times, Cerebras is well positioned to capitalize on the growing need for faster, more efficient computing. Its expanding partnerships also demonstrate how different computing architectures can work together to address the performance and economic requirements of AI at scale.

To learn more or access the Cerebras fast inference cloud, visit cerebras.ai and watch the full conversation.

‍