EpicSemi on Why Agentic AI Needs a New Unit of Compute

I’ve known Chloe Ma, Chief Business Officer of EpicSemi, since our days working together at Intel. Back then, our conversations centered on one company’s technology. On a recent In the Arena episode, we explored a much bigger question: Does agentic AI require a different approach to compute infrastructure?

EpicSemi believes it does. The company is developing a portfolio of technologies for enterprise agentic AI and sovereign AI, spanning compute, storage, interconnects, cooling and security. According to Chloe, conventional CPU-GPU architectures may not deliver the best economics for complex agentic workflows.

From Answers to Outcomes

Chloe sees agentic AI as the point where generative AI moves beyond answering questions to completing complex enterprise tasks.

She illustrated the difference with a workplace example. A manager asks for last quarter’s European sales figures. While a conventional AI system retrieves the data and provides an answer, an agentic system receives a broader assignment: determine why European sales are declining and recommend corrective action. This requires database queries, analysis, planning and coordination across systems, memory access and security controls. An agent may execute multiple steps before producing a useful result.

“It’s not just providing an answer,” Chloe said. “You have to deliver an acceptable outcome.”

This transition has significant implications for infrastructure design. Rather than optimizing individual inference requests, operators need to consider the performance, efficiency and security of entire workflows.

Putting the Sheeter in the Kitchen

Agentic AI workloads involve multiple types of processing. CPUs manage and coordinate tasks, while AI accelerators perform the intensive calculations required for inference. In conventional systems, data must travel between the CPU’s system memory and the GPU’s high-bandwidth memory (HBM). These transfers add latency and consume energy.

Chloe explained the challenge with a croissant-making analogy.

Making laminated dough requires a specialized machine called a sheeter. Imagine preparing dough in a San Francisco kitchen but keeping the sheeter in San Jose. Every time the dough needs another pass through the machine, someone has to transport it between the two cities.

EpicSemi wants to put the sheeter directly in the kitchen.

The company is developing what Chloe described as “a small, complete and fungible unit” of compute. Its chip combines general-purpose performance cores with AI cores, both based on RISC-V. The AI cores use vector and matrix extensions to deliver approximately 150 TOPS (Tera Operations Per Second) of AI performance. This allows the chip to handle both workflow coordination and AI inference without relying on separate CPU and GPU processors.

Both types of cores share system memory and storage, while the chip also incorporates enhanced security features. By keeping compute and memory resources on the same chip, EpicSemi aims to reduce data movement, latency and energy consumption.

“We believe this could be the unit of compute for agentic AI,” Chloe said.

Chloe envisions these units supporting specialized agents for financial analysis, legal review and IT operations. Because many expert models are considerably smaller than the frontier-scale alternatives, operators could keep them resident in memory and allocate compute resources through software orchestration.

Measuring Completed Tasks, Not Just Tokens

Much of traditional AI infrastructure was designed around the requirements of large-scale model training and high-throughput inference. GPU clusters remain effective for those workloads, but Chloe argues that agentic AI requires different performance metrics.

She drew a parallel with the transition from mainframes to warehouse-scale computing, when operators adopted large clusters of commodity servers to improve cost efficiency and scalability.

For agentic AI, Chloe argues that operators should measure how many tasks they can complete securely within a given power or total cost of ownership (TCO) budget.

“It’s really about how many agentic tasks that you actually securely complete within your power budget or within your TCO budget,” Chloe said.

That shifts the focus from maximizing raw compute performance to improving the efficiency of the entire workflow, including orchestration, memory access and inference.

Sovereignty Goes Beyond Data

As enterprises and nations invest in AI, control over the underlying infrastructure is becoming increasingly important.

Chloe described three dimensions of sovereignty:

1. Data sovereignty concerns control over information.
2. Infrastructure sovereignty encompasses silicon, software, security and supply chains
3. Operational sovereignty requires the expertise to deploy, manage and maintain these systems independently.

EpicSemi uses the open RISC-V instruction set architecture (ISA) to support this approach. Unlike proprietary ISAs, it allows organizations to develop compatible processors without relying on a single architecture vendor.

According to Chloe, this gives nations the freedom to choose among suppliers and potentially develop domestic silicon capabilities.

EpicSemi operates in both Santa Clara and Saudi Arabia, and sovereign AI initiatives in the latter have attracted substantial investment. Across the Gulf Cooperation Council (GCC) region, organizations are investing in infrastructure and Arabic language models to meet regional needs.

Chloe emphasized that sovereignty also depends on practical capabilities. Hardware alone cannot provide operational independence. Organizations also need the software, security capabilities, supply chain relationships and technical expertise to manage their infrastructure.

Software Will Determine Adoption

EpicSemi is working with early adopters, sovereign AI initiatives, and software partners to support deployment of its architecture. Agentic workflow orchestration is particularly important: the software must assign tasks to the appropriate processors, coordinate specialized agents and allocate resources efficiently.

“Without these type of software, our hardware would be useless,” Chloe said.

The company is also pursuing opportunities with enterprises and AI startups in financial services, investment and manufacturing, where organizations increasingly combine conventional applications with AI-driven workflows.

For EpicSemi, the next challenge is to demonstrate measurable performance and efficiency gains across representative agentic workloads while building the software ecosystem required for broader adoption.

The TechArena Take

My biggest takeaway from my conversation with Chloe is that agentic AI could change how we evaluate infrastructure performance.

Tokens per second and raw accelerator throughput remain important, but they don’t capture the full cost of completing a complex, multistep task. Operators also need to account for orchestration, data movement, security, latency and overall resource utilization.

What interests me about EpicSemi’s approach is the focus on these requirements at the architectural level. Its RISC-V design combines general-purpose and specialized compute with shared memory, while its sovereignty strategy gives customers additional choices over how and where they deploy AI infrastructure.

The next step is validation. I’ll be watching EpicSemi’s agentic benchmark results and the development of its software ecosystem. Both will be critical to determining whether smaller, flexible compute units can deliver better performance per watt and per dollar than conventional architectures for enterprise agentic AI.

Listen to the full In the Arena conversation with Chloe Ma to learn more about EpicSemi.

‍