Welcome to
the Forum

OCP Global Summit week opened for me with brunch and a question that has percolated in my mind all year: What does it take to run AI agents in production? MiTAC Computing hosted its Executive Tech Forum on Monday morning , and I had the pleasure of moderating a panel with Intel, SK hynix, Solidigm and MiTAC under the title Powering the Agentic Era: Compute, Memory, Storage and Scale.

Each company took a few minutes to make the case for the layer it builds. Assembled end to end, the four presentations delivered a comprehensive view on how innovation is accelerating across the system and the data center. Only in working together can we solve this open systems challenge that is becoming more urgent. Agentic AI changes the demand on every part of a rack or pod level platform at once, and the companies that treat the challenge as a single design problem will deliver to true operator requirements. Here is more of what I took from the room.

From Training Once to Agents All the Time

SK hynix America’s Vis Valluri, senior director of industry intelligence, set the frame with a simple progression. Training happens once. Inference runs in real time. Agentic AI runs all the time. He made his claim citing data from OpenRouter and a16z showing that agents overtook humans as the main source of token traffic early this year, with token usage up 14 times since.

The mechanics of agentic workflows explain this surge precisely. A regular inference request makes one pass through prefill and decode and returns a response. Simple and linear! An agent works in a loop: prefill, decode, call a tool, observe the result, then fold that result back into the context for another round of prefill. The KV cache grows with every turn. Valluri put the consequence in one number. More than 85% of agent tokens come from cached prompts, which means agents spend most of their time rereading context they already hold.

That single behavior sets up everything the other speakers had to say and defined a challenge across compute, memory and storage working tightly in tandem.

Compute Sits at the Foundation

Gary Brown of Intel opened his segment with a message that compute acts as a foundation to memory, storage, broader infrastructure (don’t forget the network!). As the resident CPU builder, he made the case for compute with throughput data.

Brown showed Intel Xeon 6 against competing CPU instances on healthcare agent workloads covering reasoning, security and control operations, laying out performance benchmarks of the alternative platforms. And while Intel was seeking a performance leadership win, the bigger takeaway for me is that in the agentic era, core functions of workload management matter a lot... and these are the realm of CPUs.

Intel’s own recent announcements argue that as AI turns agentic, the bottleneck moves toward orchestration, concurrency and data movement. Brown’s portfolio slide spread agentic scale across what Intel calls the intelligence and execution layers, with GPUs handling prefill. An agent spends much of its life between model calls, running tools and checking results. Someone has to schedule all of that, and the processor is a natural orchestrator.

Memory Gets New Floors

Valluri took Intel’s view and built from it with a memory centric perspective on the same challenge. SK Hynix sees the hierarchy being rebuilt with new tiers that fill the gaps between SRAM, HBM, DRAM and SSD. He laid them out from fastest to largest: a G0.5 tier of 3D-stacked DRAM with near-SRAM speed, HBM at G1, a G1.5 tier of stacked flash called HBF offering roughly 1 TB at terabit-per-second scale, conventional DRAM, a G2.5 tier of CXL-pooled memory in the multi-terabyte range, and SSD beneath it all.

The demand driver he chose was personal agents, pointing to Meta’s personal agent running on Muse Spark. This agent remembers details you mentioned once, keeps working after you close the app and runs in a dedicated cloud VM for each user. Persistent context that grows per user means more memory at every tier. When I think about the scale of that arithmetic across a large user base, the new tiers look like necessities rather than options, and the importance of memory and storage innovation argue to take center stage as the existential challenge of agentic performance.

The Price of a Prompt

So where was storage in all of this? Enter Solidigm’s Ace Stryker. He made the bluntest case of the discussion, using simple math as his tool. He started with a 14-word question: what’s a cool bird I might see when I’m in San Jose next week? That prompt is 18 tokens. The model processes far more. System prompt and policies account for 10,000 tokens, retrieval for 15,000, tool schemas for 5,000 and conversation history for 12,000. The total reaches 42,018 tokens. My guess is, in reality, we could argue it being even higher.

And... that context has to live somewhere. For one user making one request, Stryker calculated about 13.1 GB of KV cache, scaled from NVIDIA’s figure of 40 GB at 128K tokens for Llama 3 70B at FP16. Multiply by concurrent users and sessions and the memory bill scales quickly.

Solidigm’s answer to drive efficiency into the equation is to transfer that model state on SSD and restore it rather than recompute it. In work with Metrum AI, retaining KV cache on a Solidigm D7-PS1010 cut time to first token from 31 seconds to 2.8 seconds in a single-user sports archive scenario. In a separate video analytics test, pairing KV cache with DiskANN retrieval lifted queries per second by 14%, from 2,071 to 2,371 at 1,024 query threads. Stryker flagged these as separate results rather than a combined speedup, and I will repeat that caution here. He closed on a line that deserves a spot on every infrastructure budget review: every token has a resource cost, and every resource has a dollar cost.

Rack-Scale Reality

MiTAC Computing ’s Raymond Huang took the conversation from cutting edge silicon innovation to system delivery at scale, under the banner, “Where AI Power Meets Rack-Level Scalability.” Silicon only matters once it ships as a working system, and his slides were about that last mile and connecting acute operator requirements in a couture fashion with bespoke systems leveraging industry standards.

MiTAC cited more than 30 MW of rack-scale manufacturing power in 2026, capacity for 5,000 racks per month, networking up to 800 Gb/s and power readiness from 480 VAC up to 800 VDC. The MiTAC California footprint anchors this: It's Newark site features an NPI direct liquid-cooling facility, while San Jose provides dedicated liquid-cooling manufacturing. A mega-scale, six-building campus in Fremont further strengthens the company’s ability to design, validate, and produce advanced computing solutions. From PCB and motherboard design to barebone, systems and complete AI clusters, MiTAC delivers vertically integrated, in-house engineering that accelerates time to market and ensures end-to-end quality. On the product side, Huang showed a 52U high-density liquid-cooled rack, including a direct liquid-cooled MI355X GPU configuration, and a 256-GPU cluster example with in-row coolant distribution and a separate aggregation switch rack.

MiTAC pairs the hardware with open firmware and management software called MiOBMC™, MiOPF™ and MiCoreView™, and with ecosystem partners including DDN, Rafay and Canonical Ubuntu. A partnership with Tonomia targets modular AI data centers. Its Diamond Cooling approach aims to keep token generation throttle-free at standard data center temperatures, which could trim power and cooling costs if it performs as described.

The delivery model closed the loop. US-based engineering and project management, local system and rack integration, and a build-to-inventory approach aim to shorten lead times for customers who cannot wait out a long supply chain.

The TechArena Take

From soup to nuts, one word runs through the discussion as a throughline. Agents carry context from step to step, and context is state. State has to live somewhere, and every element of infrastructure is now being asked to hold more of it, closer to the processor and for longer. SK Hynix described the tiers that hold it. Solidigm priced it in both gigs and dollars. Intel claimed the baton with the statement that the processor orchestrates it. MiTAC showed how to package it all into racks that arrive on schedule.

Run the memory math once and the stakes come into focus. A single request can carry 13.1 GB of cache, and a personal agent maintains its memory between sessions. The capacity question becomes a per-user number multiplied by a sea of agents and humans. Size up that investment, and the case for re-tiering and deeper performance tier SSD use comes into acute focus.

Capacity plans built around model size and GPU count will miss the largest cost driver in agentic deployments. Plan around the workflow and sustained memory: how large does it grow per session, how many sessions run at once, and where each piece of it should live. Operators who answer those questions across compute, memory, storage and rack design together leveraging all of the standards the OCP community provides will build agent infrastructure that scales at speed.

The open hardware community’s flagship week produced a 256-CPU rack built for AI agents, a new inference accelerator, an 800-volt AI factory already under construction and a cooling paper with a blunt message for all of them: the industry adopts a new method only after the old one fails, and it pays for the delay.

At the 2026 OCP Global Summit in San Jose, the open hardware community moved its focus to readiness. The specifications, supply chains and skills for the next infrastructure transition are being built now, before a chip or a power grid makes them mandatory.

The CPU Returns to the Main Stage

The clearest sign of that shift came from the processors that feed AI accelerators rather than the accelerators themselves. Arm’s Mohamed Awad told the Monday keynote audience that AI agents, which he described as software that automates a person’s intent, now run as dedicated virtual machines with their own CPUs, memory and security isolation.

“An agent is just a virtual user. A virtual user who never sleeps,” Awad said.

Awad split the market into three CPU jobs: the head node feeding the accelerator, the cloud-native CPU serving users and managing infrastructure, and the agentic CPU acting as the virtual user. Arm published a white paper on CPU performance for agentic AI, developed with Fujitsu, Meta and OpenAI, and asked the OCP community to treat it as a starting point.

NVIDIA’s Ian Buck answered with hardware. The company’s Vera CPU rack, an MGX design holding 256 liquid-cooled CPUs, is now shipping, and NVIDIA contributed its third-generation MGX 1.2 rack architecture to OCP, down to the compute tray, busbar and quick disconnects. Buck said NVIDIA designed Vera for single-threaded speed because a gigawatt-scale factory of GPUs should not wait on a CPU.

AMD’s Lisa Su made the same point from the other direction, introducing Helion, an OCP-based rack for agentic AI that customers can configure CPU-only or with GPUs, with partner systems planned for November.

Accelerators still had their moment. Microsoft unveiled Maia 300, an inference accelerator the company says delivers up to 7x the rack-scale token throughput of Maia 200, with multi-gigawatt deployments planned from 2027.

Scale-Up Ethernet and the Life of Copper

Broadcom’s Asad Khamisy gave the summit its most direct example of an industry extending an incumbent technology rather than replacing it. Five or more hyperscalers now run Ethernet scale-up networks in production, he said, and the OCP Ethernet for Scale-Up Networking effort, ESUN, has grown to more than 175 organizations since its launch at last year’s summit.

On cabling, Khamisy said new modulation can double bandwidth inside a rack while holding copper’s 2-meter reach, which he described as extending the life of copper in the rack. Beyond a thousand XPUs, he said, operators are moving to hybrid designs with copper inside the rack and optics across racks. Microsoft’s Reynold D’Sa pointed to the Optical Compute Interconnect MSA, the first optical specification for multi-rack scale-up systems.

Power Climbs to 800 Volts

Power delivery produced four new specifications in a single week. Buck said the first 800-volt DC AI factory, built by Visionbay on the OCP standard now in development, cuts copper use by about 45% and scales from today’s 100-kilowatt racks toward 1 megawatt. Microsoft contributed its Mt. Diablo 800-volt architecture, which separates power conversion from the compute rack, to OCP at versions 0.7 and 1.0, and D’Sa said the company is working with utilities on a common grid-interface specification released last month.

Google, Meta, Microsoft and NVIDIA published an OCP Grid Resilience Specification that targets restoring near-full load within two seconds of a grid disturbance. Meta described its Prometheus cluster in Ohio, spanning more than 30 structures, and said prefabricated fabric buildings can bring capacity online in three to five months. OCP CEO George Tchaparian said the summit drew roughly 15,000 attendees with a 3,000-person waitlist, and the foundation expects more than 700 member organizations by year-end.

The Paper That Names the Pattern

Each of those announcements reads differently against “Thermal Boundaries in AI Infrastructure,” the 104-page paper OCP published on Oct. 12. Its authors, from Promersion, Oracle, Meta and Avayla, name the dynamic that governs cooling transitions: a new method is adopted only when the current one can no longer do the job, because each new method brings a dependency the ecosystem has good reason to defer. The paper calls this the least-disruption law.

The cost of that deferral has a name, too. When air cooling hit its physics limit on the hottest accelerators, the paper says, “only a thin layer of downstream vendors was ready for it,” and the supply of cold plates, manifolds and coolant distribution units fell short. The authors call the resulting bill “preparation debt.”

Single-phase cold plate has a physics limit of its own, projected near 4,000 watts for a reticle-class die, and the paper notes that frontier package power doubles roughly every two years. The authors’ answer is not a successor technology but a toolbox: air, cold plate, immersion, two-phase methods and microfluidics kept ready, interoperable and matched to the heat, the site and the moment.

The keynotes supplied the toolbox in practice. Microsoft’s third-generation Mt. Atlas heat exchanger nearly doubles cooling capacity in the same footprint, and the company now runs multiple independent water loops at different temperatures for a mixed fleet of liquid-cooled and air-cooled racks. NVIDIA’s MGX 1.2 contribution standardizes flow rates and quick disconnects so that OCP members can build compatible parts, Buck said, and he said assembly of the Vera Rubin compute tray dropped from 90 minutes to one minute.

The TechArena Take

The thermal paper closes with a line that could serve as the summit’s thesis: “Air cooling is finishing. Liquid cooling is just beginning. No single method wins, and the only real choice is to be ready for all of them.”

We read this year’s OCP Global Summit as the week the community put the paper’s advice into practice. The 800-volt specification, the grid resilience specification, ESUN and the OCI MSA all arrived before the parts that depend on them have shipped at volume. That timing is the readiness the paper asks for.  

The paper also names the gap that the keynotes left open. Commissioning, the step between building a multi-technology thermal system and trusting it with production load, now binds schedules as tightly as physics binds the chip, and the paper says no single vendor owns it. Operators should expect the next round of OCP work to land there, at the seam between the server and the building, where the paper says the lasting value sits. The companies that prepare that seam now will deploy when the next wall arrives. The rest will scramble, and the paper has already shown what the scramble costs.

The boundary organizations once drew around their data now has to extend to the systems that use it. Sovereign AI gives organizations a path to make those systems truly their own.

Hospitals, banks, universities, and national labs have spent decades building systems to protect sensitive data, with patient records, transaction histories, and research files secured in tightly controlled environments.

AI is now putting time-tested safeguards under pressure as organizations seek access to increasingly capable models.

This often requires entrusting proprietary material to third-party infrastructure: every prompt or document retrieved by an LLM can shift confidential information beyond the environment designed to protect it.

Sovereign AI is an approach to AI adoption that gives organizations control over their entire AI stack, including models, compute, software, orchestration, and deployment. This doesn’t merely impact where information resides, but which technologies enterprises can adopt, how they operate, and whether resources remain available as demand grows.

Sovereign AI requires control across the stack

Sovereign AI often starts with a straightforward requirement: sensitive data must remain on premises or within a specific jurisdiction. But keeping data local doesn’t automatically ensure autonomy.

Sovereignty is control across the full AI stack. If you are not owning some piece of it, then part of your intelligence is outsourced, and it is not sovereign to you.

That distinction matters because every external dependency introduces another point where an organization can lose agency. A hospital may rely on a provider’s roadmap to support clinical applications, while financial institutions can face changes in pricing or availability, and research organizations may encounter export restrictions that limit access to critical technology.

These dependencies extend beyond data storage. A company can keep its proprietary datasets on premises while relying on external systems to process them, models to interpret them, or suppliers to provide the underlying compute. Organizations therefore need to consider what happens after they secure the data:

• Who controls the model?
• Can teams inspect and tune it?
• Will the necessary equipment remain available as deployments expand?
• Can software investments survive a change in supplier strategy or export policy?

The same concern applies to future capacity. AI deployments rarely remain fixed, so organizations need confidence that the equipment supporting them will remain available as demand grows.

Otherwise, supply constraints can limit expansion, disrupt budgets, and affect long-term planning. For example, a deployment can satisfy residency requirements at launch and still create future dependencies if additional equipment becomes difficult to procure or critical software relies on a particular roadmap.

Sovereignty ensures control over where information resides and the infrastructure used to process it.

Sovereign AI changes the compute equation

Bringing the complete AI stack inside an organization only works if the infrastructure can support its workloads efficiently. For many organizations, that means reconsidering architectures modeled on hyperscale environments and GPU-dense racks, which were designed for enormous training and inference workloads.

Sovereign AI often looks different, particularly as teams adopt agentic applications that combine AI inference with substantial general-purpose computing.

Agents shuffle through multiple operations as they complete a task. They orchestrate workflows, call tools, enforce security policies, retrieve information, query databases, and generate tokens. CPUs handle many of the control-heavy steps, while AI accelerators execute the matrix operations behind model inference. Those demands can alternate throughout a single workflow as an agent moves between reasoning, retrieval, tool use, and model execution.

For sovereign AI environments, the workload mix directly affects infrastructure decisions. A platform optimized primarily for token generation can leave expensive accelerators waiting while orchestration, database queries, security checks, or other CPU-heavy operations constrain throughput. Adding more GPUs does not necessarily address those bottlenecks, either.

Overprovisioning accelerators also raises the cost of maintaining AI infrastructure. A sovereign deployment needs enough acceleration for model operations alongside the CPU capacity and memory bandwidth required to support the rest of the workflow.

Sizing each resource around actual application requirements can therefore improve utilization while keeping more of the workload within a controlled environment, so organizations can design around applications they intend to run, rather than reproducing a hyperscale architecture inside their own data centers.

Balancing compute around sovereign AI workloads

Sovereign AI infrastructure should enable mixed workloads without overprovisioning one type of compute. Epic Semi’s Contrail AIX, for example, combines general-purpose processing and AI acceleration within one platform.

Teams should always look to support the full agentic workflow within the infrastructure they control. CPUs can handle orchestration, branching, security, database operations, and tool calls while AI cores execute model workloads. A common fabric and memory pool keep both types of compute running in the same environment. For instance, Contrail AIX combines 32 performance cores with 16 AI cores connected by a shared on-chip fabric, allowing general-purpose and model operations to access a unified memory environment.

This balance also provides more flexibility as agentic applications evolve. Enterprises are still learning how production workloads distribute demand across general-purpose processing and AI acceleration. Rather than reproducing a hyperscale configuration by default, teams can size sovereign AI infrastructure around the applications they intend to run. This approach connects control over the AI stack with the practical ability to operate efficiently at scale.

Open architecture reduces long-term dependencies

Maintaining sovereignty requires isolated workflows, enforced policies, securely managed systems, and the ability to expand capacity without unnecessarily introducing new external dependencies, especially as architecture and market conditions change. Revised product roadmaps, export policy, or component availability can undermine existing investments.

Open architectures give enterprises more freedom over the technology behind their AI systems, permitting multiple vendors to develop compatible components around a shared instruction set. Teams subsequently have access to alternatives if one supplier can no longer meet their needs.

One example is RISC-V, an open instruction set architecture (ISA) used to design processors. Or, as Sherwani describes it: “RISC-V is owned by the world. Anybody can adopt it, build on it, and contribute back to it.” RISC-V International governs the specification from Switzerland, creating a shared architectural foundation beyond any single processor supplier or national jurisdiction.

The choice of architecture can also affect the longevity of investments. Organizations may spend years developing applications, optimizing software, and integrating systems, and continued access to the foundation is critical. An open ISA provides more flexibility when there are shifts in supplier strategies, availability, or other external conditions.

Openness, however, still needs to translate into deployable infrastructure, including virtualization, memory expansion, standard interfaces, secure management, and other critical capabilities for AI systems. In practice, that means pairing architectural openness with critical production server capabilities, such as full hardware virtualization, IOMMU support, and standard interfaces.

Enterprises also need a realistic path to procuring equipment, especially in regulated and geographically distributed markets. These considerations determine whether architectural openness actually supports meaningful autonomy.

Proven infrastructure supports long-term sovereignty

Sovereign AI simultaneously supports and depends on an organization’s ability to maintain control as deployment expands. That makes infrastructure maturity and long-term availability important considerations: a proven platform can reduce risk, simplify integration, and give enterprises a clearer path to additional capacity without introducing new dependencies as requirements evolve.

Production AI, however, relies on more than silicon. Operating systems, storage, networking, memory, management, and cooling all have to work together to support reliable deployment, operation, and expansion at scale. For enterprises, the broader ecosystem can shorten the path from evaluating a processor to running AI workloads on infrastructure they control.

Supply continuity provides another benefit: sovereign AI deployments can expand significantly over time, making future access to equipment part of the initial architecture decision. With greater operational control, organizations can integrate the technology into existing environments, scale capacity alongside workloads, and preserve their infrastructure investments over time.

AI sovereignty enables controlled scalability

The AI stack is already absorbing unprecedented institutional knowledge, decision-making and computing activity. As AI becomes more deeply integrated into institutional operations and agents handle increasingly complex workflows, controlling the infrastructure beneath it will become critical.

Sovereignty gives organizations a way to retain that authority: where workloads run, which models they use, how systems evolve, and how much capacity they add. This dynamic fundamentally changes the role of AI infrastructure. Decisions about compute, software, models, and supply directly affect how much control organizations retain over their AI operations.

An architecture designed around internal requirements can give teams greater freedom to adopt new capabilities, adjust resources as workloads develop, and make technology choices according to their priorities. That flexibility becomes more valuable as AI moves from discrete applications into workflows that organizations rely on for essential operational functions.

The benefits compound over time, enabling enterprises to advance without forfeiting the control they worked to establish in the first place.

Contrail AIX translates these principles into infrastructure enterprises can deploy. Specifically built for AI environments, the platform combines compute, networking, storage, and the supporting software stack into an integrated and scalable system, enabling organizations to own their intelligence.

‍

VEIR closed an oversubscribed $110 million Series C round to move its superconducting power delivery systems for AI data centers from technology validation to commercial deployment and scaled manufacturing.

The Woburn, Mass., company will use the capital to expand manufacturing capacity, advance its product portfolio and build the technical and commercial capabilities needed to support adoption at scale. It expects its first commercial deployments in 2027.

Matter Venture Partners and Tyche Partners co-led the round, joined by LG Technology Ventures, Gates Frontier, Sabanci Climate Ventures and Hui Capital. Existing investors including Engine Ventures and Congruent Ventures also participated. VEIR has now raised $225 million in total.

VEIR’s systems pair superconducting cables with cryogenic cooling, controls, termination units and monitoring. These systems move significantly more power through a smaller physical footprint and reduce labor costs, material requirements and installation complexity. VEIR has demonstrated a 3-megawatt system and is working with customers and industry partners on AI-factory reference designs.

“This financing gives us the resources to move from demonstrating the potential of superconducting power delivery to manufacturing and deploying systems at commercial scale,” said Tim Heidel, CEO of VEIR, in a press release.

On Sept. 28, I judged the Energy category of Yotta 2026’s Innovate Arena in Las Vegas, where Tim Carr, VEIR’s senior director of business development, pitched a cable that carries the same power as a copper cable tray in a conductor one-tenth the size.

I was delighted to see VEIR on the Innovate Arena stage at Yotta and to see this round announced so soon after.

TechArena Take

Yotta 2026 put power generation at the center of the AI buildout conversation. VEIR is working on delivery, where copper requires more labor, material and space as campuses grow. This round gives VEIR the capital to prove its systems in commercial deployments. We’ll be watching for its first installations.

‍

AI infrastructure contracts with hyperscalers can come with terms that are hard for smaller AI startups to navigate. On a recent podcast, I sat down with Ilkhom Sidikov, who works on the neocloud io.net’s IO Intelligence platform, to talk about how decentralized GPU networks are competing with hyperscalers by offering flexible, affordable options to AI innovators.

From Web3 Roots to an OpenRouter Listing

Ilkhom joined io.net in early 2024, not long after the company began turning a decentralized GPU network into something usable for AI startups and developers, an effort Ilkhom said got its start “in the age of Web3 and AI.”

He first worked on network infrastructure, and since 2025, his main project has been IO Intelligence, the vertical where io.net builds applications for AI startups, including model training, model inference, and video and image generation.

One recent milestone was being listed on OpenRouter, the LLM gateway many developers know. Io.net’s GPU network now hosts open-source models handling 2 billion to 5 billion tokens a day for OpenRouter.

Where Neoclouds Offer More Flexibility

Ilkhom said hyperscalers typically lock customers into one- to three-year contracts calibrated to protect their own margins, agreements that can be “really hard” on engineers and small AI startups. Under such contracts, a customer can end up paying for GPUs, like older 4090s, that fall behind newer architectures before the term ends. Even the AI grants that hyperscalers sometimes offer, he added, come with a catch. They typically require running on that provider’s own infrastructure.

Neoclouds, Ilkhom said, offer more room to negotiate, and io.net itself supports pay-as-you-go pricing for both GPU rental and LLM inference. Io.net’s specific advantage is how it negotiates use periods. Customers can rent a cluster for as little as 10 or 12 hours, and once they’re done, io.net shifts those same GPUs into its LLM inference infrastructure rather than leaving them idle.

“We never lose money on the idle GPUs,” Ilkhom said, adding that an idle GPU costs the same to run as one fully in use. That constant reallocation is what lets io.net offer clusters at a fraction of hyperscaler pricing.

Earning Enterprise Trust

Enterprise teams bring healthy skepticism to decentralized compute, particularly around reliability and security. Ilkhom said io.net encountered fraudsters early on who would alter a GPU’s metadata to make a cheaper card, like a 4090, pass as a far pricier H100.

Io.net’s fix was building software that profiles every hosted GPU on an hourly basis, checking whether each card meets performance benchmarks for what it claims to be.

On the network side, io.net connects nodes through virtual private networks (VPNs) to secure data in transit. It recently added a confidential inference feature that lets enterprise clients verify that every token generated, from the GPU level through the gateway to the back-end application, happens inside a secured environment.

Engineering for Scale and Profit

Running inference profitably at scale comes down to a handful of techniques, Ilkhom said: quantizing models down to formats like FP8 or INT4, using key-value (KV) caching to speed up throughput, and batching inference requests. Skip those optimizations, he said, and providers selling API tokens can end up burning cash, since deploying a raw model with default settings can make it hard to even break even.

Agents Raise New Demands

The rise of AI agents presents new demands on compute and data. Context windows have grown enormous, Ilkhom said, citing Anthropic’s own claim that nothing stops the company from building a model with a million-token context size. Combined with the development of MCP servers and skills that let agents pull real-time data instead of relying on static databases or vector search, agents now behave less like single inference calls and more like systems that maintain memory and reach out for information as they need it.

The TechArena Take

My conversation with Ilkhom conversation showed how decentralized providers like io.net are winning through an intensive focus on optimizing their resources. Io.net does this by keeping GPUs constantly busy rather than allowing them to sit idle between contracts.

Every technique he described, from shifting idle clusters into LLM inference to quantizing models, squeezes more usable compute out of the same hardware hyperscalers rely on too. Hourly hardware benchmarking helps build the trust enterprise customers need before relying on that hardware at all.

Ilkhom’s 5-year outlook does not predict that decentralized compute will displace the hyperscalers. After all, regulated and well-capitalized buyers have reasons beyond price to stay where they are. What his outlook does suggest is a market potentially splitting by workload, with contract-scale commitments on one side and short-duration, price-driven capacity on the other. For technology leaders, this opens up new possibilities to start matching each workload to the economics that fit it.

To learn more, visit io.net.

‍

Niklas Panten and his team at etalytics keep track of how much energy their software saves per customer and site across global deployments. Watching the numbers climb unifies the team around the company’s purpose, Panten said.

Founded in 2020 as a spinoff of Niklas’ PhD research at the Technical University of Darmstadt, etalytics builds AI software that orchestrates the cooling, heating and power systems in data centers and factories so they use less energy.

The ETA Factory

Panten, co-founder and CEO of etalytics, built his PhD on a mix of mechanical engineering, computer science and energy science. He helped develop the ETA Factory, a full-scale research factory TU Darmstadt opened in 2016 to test energy efficiency on working production lines.

His research group spent more than 10 years analyzing and optimizing energy systems in manufacturing plants. Its partners included Bosch, Volkswagen and Mercedes, along with chemical and pharmaceutical companies. The researchers used AI to run cooling, heating, ventilation, on-site generation and batteries more efficiently, work Panten said began before the AI boom.

“In our research, we found almost 50% in savings in those systems,” he said.

Two research projects at the factory, PHI-Factory and SynErgie, ran from 2014 to 2018 and produced the first ideas and prototypes of etaONE, the platform etalytics sells today, according to the company. Thomas Weber, who co-founded etalytics with Panten and is now its chief strategy officer, worked on his PhD in the same lab. The founding team, which includes CTO Björn König, came together in 2019.

“(This work) came from personal passion; I was a controls engineer and I started on power converters,” he said. “I found out that we can… make (power converters) much more efficient. But even if you have the most efficient power converter, if you have the component that has a power converter in there, and it’s not operating efficiently, you’re still losing energy.”

The same problem repeated one level up. “And then looking at the component, you can optimize the component, but if it’s not optimized or orchestrated in the system, then you’re still losing energy,” he said.

That finding is the basis of etalytics: to run at its most efficient point, a whole system has to be orchestrated. In a data center, Panten applies it to cooling “from the rooftop to the server room.”

From Factories to Data Centers

In 2020, Equinix asked the team to take its approach to data centers.

“They basically asked us, ‘Hey guys, we’ve seen your potential here, can you transfer this to data centers?’” Panten said.

At Equinix’s FR6 data center in Frankfurt, etaONE cut the energy used for cooling by 900 MWh a year and avoided 240 tonnes of CO₂ emissions a year, with a payback of less than 12 months, according to etalytics.

The company kept its industrial work. At Merck’s headquarters in Darmstadt, etaONE cut the electricity used for cooling by an average of 21%, in a project coordinated by TU Darmstadt’s ETA research group. At the Stellantis paintshop in Rüsselsheim, Germany, it cut ventilation energy use by more than 60%, in a project with the university.

“Cooling is a cross-sectional technology,” Panten said. “So, the same cooling equipment that we deploy for data centers, you will find in automotive, chemical, pharma, logistics, especially cold logistics, food processing.”

Proof at Telehouse

The company’s etaONE platform sits on top of a data center’s building management system (BMS). It streams the cooling plant’s telemetry into a physics-based digital twin of the equipment, calculates new set points every two to three minutes and writes them back to the BMS.

“We don’t rip and replace anything, we just add on top,” Panten said.

At Telehouse’s Frankfurt campus, digital twins of the cooling towers showed their performance declining. etalytics traced the decline to fouling, a buildup of deposits on the towers’ heat exchangers that reduces how well they shed heat. A heat wave was approaching, and Telehouse moved its maintenance up. Panten said the predictive maintenance product grew out of checking etalytics’ own models against the plant.

“As soon as we saw deviation, we knew it’s not our models that deviate, it’s the physics that change,” he said.

Over an assessment from April 14 to Aug. 5, 2026, etaONE adjusted the cooling tower outlet temperature and the recooling pump flow. One unit used 10.4% less electricity, and etalytics projects 497 MWh in annual savings across five units. The project won the Infrastructure Sustainability Award at the Platform Global Awards 2026.

Panten said etalytics chose physics-based models over reinforcement learning so operators know why the system makes each change.

Megawatts for Compute

Panten said energy efficiency is crucial in an industry constrained by power.

“Everyone is looking for more megawatts, so when you make megawatts available from cooling, you can put them in compute,” he said.

He pointed to research from the International Energy Agency, which projects that well-documented AI use cases could save more than 13 exajoules of energy by 2035, equivalent to 3% of global final energy consumption, if barriers to wider adoption are overcome.

Looking Ahead

etalytics opened an office in the San Francisco Bay Area in May 2026, and Panten is now based in Menlo Park, California.

Microsoft’s venture fund, M12, led an investment in etalytics in 2025. This year was the company’s first time attending Yotta, and Panten said it will have a booth next year. In October, the company plans to launch etaONE IQ, an assistant for operators’ daily work, at an event with Microsoft and NVIDIA during San Francisco Tech Week. Panten said etalytics’ AI will also move onto the coolant distribution units that serve AI factories.

TechArena Take

We see cooling orchestration as one of the fastest ways for operators to free megawatts for compute without waiting on the grid. The physics-based approach matters as much as the savings, because operators of mission-critical sites need to understand every change an AI makes before they trust it.

Customers are signing five-year contracts for AI silicon that has barely started shipping, while the models running on that hardware turn over in weeks, leaders from Crusoe, Microsoft and VAST Data said during a panel at AI Infra Summit 2026.

I had the delightful opportunity to moderate “Under the Hood of AI Infrastructure: What Top Leaders Do Differently,” with Erwan Menard, Crusoe’s senior vice president of product management; Donald Thompson, a distinguished engineer at Microsoft; and Alon Horev, co-founder and chief technology officer at VAST Data.

Across the hour, the three described an industry committing capital years ahead of knowing what it will run. Hardware gets committed in multiyear blocks and new power takes years to bring online, while the models and workloads running on them can change in weeks.

Erwan described clocks that do not agree: “I think in days. The data center guys used to think in years. Now, at places like Crusoe, they think in quarters. The energy world thinks in multiple years.”

One of Alon’s customers moves faster still, retiring the model its users are talking to every 30 minutes and publishing a newer one so that results stay fresh.

Alon called AI the “reset event of infrastructure globally,” and the technology hasn’t stopped resetting. Four themes from our discussion show where: Training and inference have folded into one continuous loop, the pace of innovation has turned into a design constraint, the megawatt has become the unit to plan around, and the data center is becoming a single computer spread across a building.

The Model That Changes Every 30 Minutes

Inference is probably bigger than training across the worldwide GPU fleet, Erwan said, and at the AI-native companies Crusoe works with, it “becomes the lion’s share” once a product reaches volume.

Open models drive a lot of inference traffic too, he said. Crusoe helps customers modify a model with their own data, put it into production and retire it on their own schedule.

“The big motivation we hear from customers is control,” he said.

Donald carried the idea further. Digital workers are trained by people inside a specific work context, and their trajectories have to improve over time, which rules out pure training and pure inference.

“You have to be in a constant learning mode,” he said.

He described sleep-time compute, fine-tuning during quiet hours, and reinforcement learning in eval environments built to replay what an agent has just gone through.

Alon’s 30-minute customer is running that pattern in production. They kept pressing his team to speed up model loading, and he asked why they needed it hundreds or even thousands of times a day.

“Because we train nonstop,” they said.

Innovation Outruns the Build Cycle

Erwan told a story about an inference engine. Someone at the show that morning claimed bragging rights for building the first one three years ago, and Erwan doubted it was still in production.

“The pace at which these things are evolving is very annoying,” he said.

Crusoe shipped its first inference service a year ago and still tunes it weekly against fresh open source, new papers and new practices. Managed Kubernetes taught him the same lesson. A general-purpose version takes thousands of engineers and years; an AI-specific one has to be excellent at a short list of things, and Crusoe’s now powers 70% of its compute fleet.

Erwan described the new pace: Research reaches production in weeks, instead of two years, and his researchers talk directly to customers, who have seen Jensen Huang’s Pareto curve and are weighing throughput against latency.

“You need to walk with them and innovate with them on the fly,” he said.

Alon agreed that standards will likely hold you back at this pace, though he sees new ones forming organically, with SNIA and various companies working together. What changed, he said, is that buyers are less afraid of proprietary technology. Before AI, buyers wanted every layer of the stack to be generic, which made it hard to innovate or change big parts of it. “And AI changed that mindset at least for now.”

Getting More From Each Megawatt

Most AI infrastructure gets bought in multiyear reservations now, Erwan said. He described five-year commitments for Vera Rubin, silicon that is only now starting to arrive. Buying that far ahead means committing to power before anyone knows what the workload will draw, and in reserved clusters already running, consumption sometimes lands well below what’s available. That gap opens room for a provider who understands both the machine’s power needs and the workload.

Closing the gap eventually means packing workloads onto shared hardware, in multitenant environments that absorb the peaks and valleys of inference traffic without stranding costly GPUs, he said. He predicted that by 2028, everyone will be optimizing for availability of megawatts.

“The energy wall is coming, and it’s going to take a lot of time to remove it,” he said.

Donald said Microsoft is working at the application and platform level to drive 24/7 inference, which is what pushes demand for megawatts.

Alon came at the same problem from the data side. He described the job of infrastructure builders as making “every megawatt, every terabyte on SSD and every CPU” serve training, fine-tuning and inference alike. He later asked whether reducing the doubled hardware some organizations carry on every stored byte could cut power along with it.

Packing workloads better only goes so far. Erwan expects power to stay short, since adjusting energy supply to demand takes years. Crusoe’s other answer runs through geography: hydro in Norway, geothermal in Iceland, and a small modular reactor paired with a modular data center that he hopes to put in service by mid-2027.

The Data Center as a Single Computer

Alon called inference “a clustered problem,” with prefill and decode disaggregated so that “some GPUs do this; other GPUs do that.” Every open session ties up a block of GPU memory for as long as it lasts. That block is the KV cache. In some models, contexts of a million tokens push it to “tens or even hundreds of gigabytes,” crowding out other users, so it gets moved to a storage system or an SSD on another host. Routing then has to send that user back to whichever machine is holding their cache, so no single GPU completes the job alone.

“Everything is integrated into one logical computer that’s doing inference, … and KV cache is becoming a core component of it,” Alon said.

Eighteen to 24 months ago, Crusoe sized its data centers at four to five terabytes of storage per GPU, Erwan said. Today it is north of 10, and a given inference proof of concept takes roughly 20% fewer GPUs than a year ago.

He also wants the agent’s memory and context treated as its own layer of the stack. There is too much of it to hold in expensive memory, so some has to move to cheaper storage, and he sees “true infrastructural challenges in delivering that context.”

We should look at a data center now, Alon said at the end of the hour, as “a single computer just distributed across so many different machines and components.”

The TechArena Take

Hardware and power get committed for years. The models running on them change in weeks. That is the mismatch, and the panel treated it as a purchasing problem. Most of their answers were things to buy: longer silicon reservations, sites in Norway and Iceland, a reactor, more megawatts by 2028. One answer was operational, packing more work onto hardware already installed and already paid for, and it got less time.

That ordering is backward for anyone who has already signed. A five-year contract cannot be renegotiated into a different clock. What can still change is how the building runs: what gets packed where, what waits for a cheaper hour, what the machines actually draw against what the spec promised.

Donald raised this on stage, asking how intelligence gets applied to running the data center itself instead of leaving a person at a dashboard. Nobody answered. That is the question TechArena would put to vendors this year: What does the cluster do on its own when the model changes in six weeks? The answer may decide who is still competitive in 2028, when Erwan expects megawatt availability to be what everyone optimizes.

For decades, the industry has named walls: the memory wall, the compute wall, the network wall, the cooling wall, the data wall. The list keeps growing. This week, more than 6,000 people converged on Las Vegas for Yotta 2026 and unpacked the wall beneath all of them: power. Without power, none of it functions.

Across three days, the builders, suppliers and founders on Yotta’s stages and on the expo hall floor made the case: The industry can deploy only as much AI as it can power, so every gain in compute, cooling, storage, networking and I/O now counts in tokens per second per watt.

The Power Gap

Varun Sivaram, CEO and founder of Emerald AI, told the Crusoe Main Stage audience on Tuesday morning how much of the AI factory capacity planned in the United States will lack grid power to run it.

“If I put some numbers to it, between now and 2028, the United States seeks to connect something like 70 gigawatts of AI factories,” he said. Less than half of that will successfully plug into the grid, he added. “There's a massive supply-demand gap just on the power side.”  

Sarfraz Taj, vice president of business development at Terrestrial Energy, opened his Innovate Arena pitch on Monday with the total capacity of data center projects now planned in the United States.

“So, Vegas is all about numbers. Let me start with some numbers: 300 gigawatts. That’s the capacity of data center projects currently planned in the U.S.,” Taj said.

The first response to the gap is already used up, he explained. Some hyperscalers and developers went directly to power producers, restarted shuttered nuclear plants and pushed existing plants to higher output.

“Against a 300 gigawatt plant capacity for data centers, that low-hanging fruit is now all but virtually gone, so not a viable way to get additional electricity,” he said.

Chris Lander, vice president of XFRA at Span.io, described how long the shortage of grid connections now adds to a data center build. He listed interconnect queue delays, transmission line build-outs, lead times for substations and transformers, and public opposition in local communities.

“A year ago, it would take around three years to build a data center for AI,” he said. “We’re seeing that three years already extend to five years.”

Solar Power at Night, From Orbit

The farthest source of new power pitched at Yotta sits in orbit. Abdullah Al-Shakarchi, co-CEO of Overview Energy, presented it in the Innovate Arena on Monday.

“We are in the business of selling invisible light,” Al-Shakarchi said.

Overview Energy places satellites in geosynchronous orbit, where they receive sunlight around the clock. The satellites convert sunlight to electricity and use it to power near-infrared lasers. The lasers send an invisible beam to solar farms on the ground, so those farms produce power at night.

Al-Shakarchi said a few hundred satellites could power a single 300- to 400-megawatt training data center. The satellites can also redirect their beams. A satellite serving Las Vegas could send its energy to Morocco within minutes, he said.

“This is the world’s only movable power plant,” he added.

His company’s approach turns solar from a source that produces power 35% of the time into one that produces 70% to 90% of the time, using solar farms that already exist. Meta, one of the company’s first customers, announced a gigawatt agreement with Overview Energy in April to power its data centers from space. Al-Shakarchi said the company plans to reach orbit next year, power a U.S. utility-scale solar project in 2028 and serve its first customers in 2029.

Trevor Smith, CEO of Atomic-6, pitched a different use of orbit: data centers that operate in space. Atomic-6 builds solar arrays, radiators and debris shielding for those satellites. Smith said orbital data centers will stay small, in nodes of 100 to 200 kilowatts that connect to one another.

“You’re not going to see Costco-sized data centers flying around in space,” he said. “It just doesn’t make sense.”  

The Grid Already Built

Vladimir Troy, vice president of AI infrastructure at NVIDIA, said the nearest source of new power is grid capacity the country has already built. Emerald AI’s Sivaram interviewed Troy on Tuesday’s main stage. Troy said utilities build the grid to meet peak demand, and demand stays below that peak most of the time.

“Like 99% of the time, (it’s) just sitting there and being underutilized,” Troy said.

An AI factory that reduces its power draw during peak hours can use that capacity, he added. Sivaram said the U.S. grid has 100 gigawatts of available capacity today for AI factories that can operate that way. Emerald AI builds the software that lets AI factories adjust their power use. NVIDIA, Google and Emerald AI launched the AI Energy Management Alliance on Sept. 17 to advance that approach.

New Power on Site

Sites that cannot wait for a grid connection need power generated and delivered on the campus itself. Founders and suppliers at Yotta pitched both.

Emerson Reiter, chief commercial officer and co-founder of Teragen Energy, pitched fuel cells. Teragen builds one-megawatt units that fit in a 20-foot shipping container.

“Primarily, fuel cells turn molecules to megawatts without combustion,” Reiter said.

Teragen’s cells can ramp from zero to full power in minutes rather than hours, he added. The company plans commercial deployments in 2028.

Vince Canino, president and CEO of Capstone Energy+, told TechArena that the company's gas turbines can serve a data center from groundbreaking to full operation. Each turbine has a single moving part that rides on a cushion of air, so it runs without oil, coolants or water, he said.

"Because we can move our equipment around very easily, we can start out as construction power," Canino said. "As construction is done, we can move the equipment to bridge power, and then we can put it in its final permanent position as primary power to the data center."

Constantin Eis, CEO of CMBlu Energy, pitched a water-based organic flow battery that cannot catch fire. He compared the U.S. market to Germany, his home market, which is closing power plants.

“Actually, I think the capital is there, the land is there, customers are there. The only problem which is missing is the energy,” he said.

Power generated on site still has to reach the racks. Tim Carr, senior director of business development at Veir, pitched a superconducting cable that carries the same power as a copper cable tray in a conductor one-tenth the size.

“We believe that power delivery, not just power generation, is the next necessary evolution,” he said.

Melton Chang, executive vice president of power systems at Schneider Electric, told TechArena that the company is replacing the wiring and custom hardware in medium-voltage switchgear with software. He said the change shortens time to power and moves most commissioning from the job site to the factory.

“Software-defined is not just about one product,” he said. “Software-defined is really about the system.”

More Tokens From Each Watt

Once power reaches a site, the next question is how much of it becomes tokens. Peter Panfil, vice president of power architecture and application at Vertiv, said a 100-megawatt site running at industry-norm efficiency puts about 60 megawatts to productive work at the GPU. He also cited a GPU utilization rate of about 70% that he heard three times at the conference.

“That’s a pretty damn expensive sports car to only be able to run at 70% of its capacity,” Panfil said during the Tokens Per Watt panel on Wednesday, referring to GPU capacity left unused.

Sean Corcoran, vice president of sales for PowerOne at Airsys, discussed a metric called PCE: the IT load a site provisions divided by the building load it provisions.

“It’s all about finding that waste of power and efficiently using it to compute more and more,” he said.

Power spent on cooling produces no tokens. Matthew Orcutt, vice president of strategy and systems product management at Trane Technologies, told TechArena that NVIDIA’s support for 45°C facility water lets a data center cool its GPUs with dry coolers, without chillers or compressors, for most hours of the year. LiquidStack, a Trane Technologies company, launched a coolant distribution unit at Yotta that supports that temperature. Orcutt said the amount of water a data center uses is a separate design decision.

Storage Joins the Math

Storage design also affects how much work each watt produces. On TechArena’s Tuesday panel, Angel Camacho, director of product marketing at Western Digital, said the industry first answered shortages by buying more storage and more power.

“Until suddenly, we hit a wall. It becomes too expensive,” he said.

Marc Austin, CEO and co-founder of Hedgehog, described a customer demonstration at Computex. One GPU ran an inference workload in 90 seconds. The same GPU, paired with a second accelerator, ran it in 30 seconds.

"Do the kind of simple math on that and it’s six times the tokens per watt, which is huge,” he said.

Betsy Chernoff, principal AI product marketing manager at WEKA, said storage has joined that calculation only recently.

“When you get over to the inference side of the house, you can have a fully utilized GPU, and still have it incredibly inefficient,” she said.

The Side Yard

Some companies took the opposite approach and moved the compute to places where power already exists. Span, the smart electrical panel company Lander represented, places liquid-cooled inference nodes in homeowners’ side yards, next to the electrical panel. The nodes run on capacity the home does not use, and the homeowner receives subsidized energy and internet in return.

“Instead of waiting for power and traditional data center infrastructure, organizations are increasingly looking at how they can take the compute to the power,” said Angie Keeler, co-founder and CEO of Zella DC.

Submer Group launched Corenix, which builds and tests complete data center modules in a factory before shipping them to a site.

The Neighbors

Plans to build on the ground require approval from the community around the site. Niklas Panten, co-founder and CEO of etalytics, whose software reduces the power that data center cooling plants use, said he understands the objection.

“To be honest, I don’t want (a data center) in my backyard either," he said.

Allyson Klein, founder and CEO of TechArena, moderated Wednesday’s Tokens per Watt panel and gave its final thought.

“I think that we’re going to see the focus on manageability, and how we create a control plane for operators that not just takes into account the compute infrastructure, but also storage and networking, as well as potential discussions about control of your power and cooling equipment from an integrated pane of glass,” she said. “I think that those things can provide us an ability to drive up every metric that was talked about on stage today and will bring this industry closer together. What I’m excited about is actually being here to see it all happen… This industry is incredible at innovating itself out of a problem.”

TechArena Take

We came away from Yotta 2026 seeing the power wall as a design problem as much as a supply problem. The speakers with the most concrete answers treated power, cooling, compute, storage and networking as one system, and they measured that system by the tokens it delivers per watt. Overview Energy took the other approach: it adds new hours of output to solar farms that already exist. We expect both approaches to grow.

We see tokens per second per watt as the measure to watch. Operators who design their sites around it will add AI capacity in the next year without waiting for a place in the interconnect queue. The operators who win community approval will be the ones who can show their neighbors a stronger grid and lower rates.

‍

Why enterprise infrastructure must optimize the entire workflow, not only model inference

A recent study of five representative agentic AI workloads found that the dominant bottleneck shifted from retrieval to code execution, scientific tools, summarization, and model inference. In several tests, work outside the model consumed more than 80% of end-to-end latency.

That matters because an AI agent does much more than running a model. It plans, retrieves data, calls tools, updates its state and repeats the process until the task is complete. Each stage stresses a different part of the system.

For banks, hospitals, universities and national labs that buy and run their own infrastructure to keep data on premises, optimizing only the accelerator may only yield very limited impact on the entire workflow, or even worse, leave your most expensive and power-hungry accelerator waiting.

The bottleneck moves

The study, “Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective,” published in November 2025, instrumented the five workloads end-to-end across two systemsi. The results do not converge. In each workload, a different stage consumed the largest share of end-to-end latency:

Selected results from two systems and multiple benchmarks. The dominant stage changes with workload inputs and hardware configuration.

• Only in the last case is the bottleneck the matrix math the industry has been building for.

For enterprise infrastructure teams, that ratio changes the buying decision. The bottleneck moves with the work, and a production deployment runs several kinds of work at once. No single resource can be scaled up to fit the whole fleet. Take the retrieval-augmented pipeline from the study, where nearest-neighbor search accounted for more than 80% of the time and runs on CPU cores. Buying more accelerators speeds up the model inference in the remaining share. The search step, which decides how long the whole job takes, does not get faster at all.

One agent needs several kinds of compute

Look one layer down and the same pattern holds. At the model level, prompt prefill benefits from high parallel compute throughput, while token-by-token decode depends more heavily on memory bandwidth and capacity. Outside the model, data processing, tool calls, orchestration and security run efficiently on general-purpose cores. An agent moves between these phases continuously.

Balanced systems are becoming a mainstream design priority. Intel CEO Lip-Bu Tan told the Computex 2026 audience that agentic AI is “returning the CPU to a position of prominence,”ii and AMD made the same case for CPUs in the AI data center earlier in the year.iii The single-silicon era of AI infrastructure is closing.

Why we built Contrail AIX

Organizations and nations that desire to gain greater control over how AI is deployed, customized and governed keep their data on premises, deploy open-weight models and build infrastructure for the complete agent workflow deployable inside their own walls. That is what Epic Semi designed Contrail AIX to be.

What we built to run AI agents

Contrail Compute, the RISC-V CPU and AI superchip inside our Contrail AIX platform, puts 32 high-performance RISC-V cores and 16 RISC-V AI acceleration cores on one die, sharing one memory and storage space across a common on-chip fabric.

This architecture keeps retrieval, orchestration, model execution, and tool calls close to the same agent state, reducing unnecessary data movement between separate devices. Each Contrail Compute chip can serve as a native agentic AI execution unit, and these units are small, complete, and fungible. Small means granular enough to scale independently. Complete means useful work can run locally. Fungible means any unit can be scheduled, replicated, drained, or replaced through a common operating model.

Agent-aware orchestration compiles the workflow into a physical plan, places each fragment on the right resources, preserves data locality, and continuously resizes the fleet. Units pool when demand rises, fail over automatically, and disappear when no longer needed.

We built it on RISC-V so a customer’s software investment outlives any single vendor’s roadmap and any single government’s export policy. Contrail AIX carries a 5A992C export classification, which reaches substantially all markets. It supports the RISC-V hypervisor extension with full hardware virtualization and IOMMU, RVA23-compatible architecture, trusted secure boot and full-chip security management. Ubuntu came up shortly after first silicon, and llama.cpp and open-weight models are running today. Compute is the first product on a multi-generation roadmap, from a team that has ramped millions of microprocessors.

Agentic benchmarking is underway. We will publish performance results after optimization and validation so customers can evaluate defensible end-to-end measurements rather than synthetic claims.

If you operate AI on premises and want control of your data, bring us your workflow bottlenecks. We’ll help you measure them end-to-end on Contrail AIX.
‍

[1] Ritik Raj, Souvik Kundu, Ishita Vohra, Hong Wang and Tushar Krishna, "Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective," arXiv:2511.00739, submitted Nov. 1, 2025, revised April 16, 2026, https://arxiv.org/abs/2511.00739.

[1] Intel Newsroom, "Computex 2026: An Intelligent World Built on Silicon," June 2026, https://newsroom.intel.com/artificial-intelligence/computex-2026-an-intelligent-world-built-on-silicon. Quote as reported in Divyanshi Sharma, "Computex 2026: Intel announces Xeon 6+ processors, says AI will make CPUs important again," Digit, June 2, 2026,

[1] AMD, "Agentic AI Brings New Attention to CPUs in the AI Data Center," AMD corporate blog, March 13, 2026, https://amd.com/en/blogs/2026/agentic-ai-brings-new-attention-to-cpus-in-the-ai-data.html.

‍

VAST Data today introduced VAST DataEnclave, a confidential AI runtime that lets model builders place proprietary models inside a customer's own data center and run them against regulated data. The capability, built on NVIDIA Confidential Computing, previews today and ships in Q1 2027 through VAST and OEM partners including Cisco and Supermicro.

DataEnclave keeps data and model weights encrypted while they are in use, and releases decryption keys only after it verifies the environment those assets are about to enter. That gives an enterprise a way to apply a frontier model to data it cannot move, and gives a model builder a way to deploy weights into infrastructure it does not own.

The Gap at Execution

Encryption already covers model weights on storage and across the network. The gap opens at execution. To serve a prompt, weights decrypt into GPU memory, where the infrastructure's administrators, a compromised hypervisor, other tenants sharing that memory and firmware exploits can all reach them.

Enterprises and model builders each give something up to that exposure. Banks, hospitals and government agencies hold data they cannot ship to a hosted AI service, so the best models never see it. Model builders treat base weights as their most valuable asset and keep them on a short list of trusted platforms, which puts air-gapped and sovereign facilities out of reach. VAST, citing Synergy Research, puts about 50% of the world's data somewhere frontier models cannot go today.

Verify First, Decrypt Second

DataEnclave adds a secure container runtime and an attestation service inside the VAST DataEngine.

Hardware-isolated execution runs workloads in confidential virtual machines and containers, using trusted execution environments from Intel, AMD and NVIDIA Vera to encrypt guest memory, GPU memory and NVLink traffic. Active data and models stay isolated from infrastructure administrators and from other tenants on the same hardware.

Verify-before-decrypt attestation checks the trusted execution environment, including NVIDIA GPU attestation, before any decryption key is released. VAST states the rule plainly in its briefing: "No attestation. No access."

Independent key control lets each party hold its own keys in its own trust domain through bring-your-own KMS integrations. The model builder protects base weights. The enterprise protects its data and the fine-tuned weights that increasingly carry its own IP.

That key control is what VAST says separates DataEnclave from the confidential VMs cloud providers already sell. The weight owner verifies the platform itself, keeps its own keys and gets the same arrangement whether the deployment sits on-premises, in a cloud or behind an air gap.

VAST logs attestation events, key releases and enclave lifecycle actions to a tamper-proof audit trail in the VAST DataBase. Deployments run connected or fully air-gapped, on the open CNCF Trustee stack or on Fortanix confidential AI infrastructure. The same runtime gives agents isolated sandboxes through VAST AgentEngine.

More Than 20 Partners at Launch

Cohere, CrowdStrike, Deepgram, Factory, Fundamental, NVIDIA and TwelveLabs are among the model builders signed on. Cisco and Supermicro will deliver integrated confidential AI systems. BUZZ HPC, Nscale and Sharon AI plan to offer attested sovereign environments, and Fortanix supplies the attestation and key management layer.

Fundamental sells NEXUS, a large tabular model it says runs inside a bank's or health system's own infrastructure.

"Confidential computing turns trust from a promise into a proof, and that's the difference between AI that regulated industries can pilot and AI they can actually put into production," said Jeremy Fraenkel, CEO of Fundamental.

Sovereign clouds gain a second benefit. Hardware enforces the isolation, so a regional provider can host proprietary models and regulated data without dedicating a whole machine to each tenant.

Models as a Managed Resource

DataEnclave also advances a larger VAST position. Models belong under the operating system rather than on top of it, managed as a logical resource alongside data and paired to tasks by purpose, cost and security constraints.

"Models are becoming a resource the operating system has to manage, the same way it manages data," said Renen Hallak, founder and CEO of VAST Data. "That means knowing which model fits which task, what it can see, who can use it and under what rules, and doing all of that inside the same security and operational boundaries an enterprise applies to everything else."

TechArena Take

The most interesting part of this launch sits one level above the enclave. VAST is treating models the way it treats data, as a resource the operating system manages, and DataEnclave turns that idea into an enforceable policy about where a model runs and who holds the keys.  

Operators running more than a handful of models across their environments should look closely at what it means to govern models with the same controls they already apply to data. The detail worth the most attention here is key control over fine-tuned weights. Those weights carry what an organization has learned about its own business, and many teams still treat them as a byproduct of training. Giving enterprises the same protection the frontier labs hold for their base weights sets a precedent we expect the rest of the market to follow.

The partner list tells its own story. Seven model builders agreed to send their models into environments they do not control, which points to where enterprise AI is heading. Models travel to the data. For financial services, healthcare and government, that direction of travel opens use cases that have been waiting on it.

We will be watching two things through 2027: the first deployments in regulated sectors, and the performance envelope operators get with attestation and enclave execution in the path.

‍

AI infrastructure leaders from around the globe gathered this week for the 2026 AI Infra Summit in Santa Clara against a backdrop of AI model slowdown discussions. Speakers from Crusoe to OpenAI separated the two issues: the slowdown call concerns frontier model training, and the infrastructure they are building runs inference.

Across three days at the summit, operators and suppliers pointed to the continued growth of inference demand, stating that they can build only as fast as they can secure power. They laid out how to get more tokens from each megawatt through rack-level power management, campuses that ramp with the grid, and cooling that uses no water.

The Slowdown Question

Ed Nelson, co-founder of the summit, put the question to Lip-Bu Tan, CEO of Intel, on the main stage: does the slowdown call mean the infrastructure buildout has to pause, or does it shift the focus toward inference at scale? Tan responded by saying he’s seen multiple cycles of technology growth, but this one is bigger.

“…Some correction is kind of natural, and I don't overreact to it,” Tan said. “But something that you have to pay attention (to): We live in the Silicon Valley. It’s a bubble. And we all get so excited about AI. When you go to a different part of the country…you find that there's a very strong anti-AI sentiment.”

Tan said leaders in the industry have to address the negative side directly while continuing to adopt the technology, prioritizing security and ensuring that data and privacy are not compromised.

The Workload Changed

Ian Buck, VP of hyperscale and HPC computing at NVIDIA, put numbers on how much the inference workload has changed. The 2023 chat workload that became the industry benchmark ran about 1,000 input tokens, a 4,000-token cache and about three turns, with a person reading each answer before asking the next question. The agentic benchmark Buck cited averages 142,000 input tokens. He described agents running 65 turns to reach an answer. Commercial services already run thousands of sub-agents, he said. He called the new workload 100 times more demanding.

“Agents have now taken humans out of the loop,” Buck said, “and the compute will be consumed as fast as the agent can make the turn.”

Demand is not slowing

Dave Patterson, distinguished engineer and Google Fellow, described hardware staying in service long past its amortization schedule inside Google. TPU v2 and v3, chips from eight and ten years ago, still run inference. Google amortizes accelerators over six years and CPUs over eight, and Patterson said the hardware will last longer than that.

“There’s just such a desperate need for cycles that people are absolutely keeping them beyond the amortization,” Patterson said.

Bloomberg’s Dina Bass asked Sachin Katti, VP of compute strategy and GPT-Infra at OpenAI, whether the safety and alignment discussion affects OpenAI’s compute plans. He said it increases the need for compute. Making very capable models safe requires safety models and alignment models. Those have to be trained and served as well.

“If anything, I think it actually double underscores the need for more compute,” Katti said.

During a TechArena panel, an audience member asked when the industry gets its next moment to pause and reset expectations. Erwan Menard, SVP of product management at Crusoe, compared the current supply imbalance to the flooding of hard disk drive factories in Thailand after a monsoon in 2011; at the time, Thailand manufactured ~40% of the world's hard disk drives. He called it the closest precedent in his more than 25 years in infrastructure.

“What we’re experiencing now is way bigger than that at every layer of it, which tells us that demand is essentially insatiable,” he said.

Two years ago, Menard said, the question at a session like this would have been what a provider planned to do with the Hopper GPUs on its balance sheet. NVIDIA was shipping a new generation every year, and there was no time to monetize the old silicon before the amortization schedule ran out. Two generations later, the GPU-hour price of a Hopper cluster has risen since the beginning of the year. Menard called that another validation of demand.

Donald Thompson, distinguished engineer at Microsoft, took the audience question after Menard.

“A lot of it is irrational,” Thompson said. “And I think it’ll take an exogenous event to create a pause. I don’t think that the market itself will pause. Even if it’s in our best interest to pause, I don’t think we will self-regulate in that way.”

Asked what kind of event, he said geopolitical.

Inference is Production

John Roese, global CTO and chief AI officer at Dell, described what it took to get AI into production inside his own company. Dell had 900 AI projects when he took the role. The company discarded them and restarted with four strategic projects covering sales, services, supply chain and engineering. Roese put the result at $30 billion in revenue growth over two years alongside a decline of about 7% in absolute cost.

“To get an enterprise into production is non-trivial,” he said. “Many of the things you have to deal with have nothing to do with the technology.”

Power is the Constraint

NVIDIA’s Ian Buck said the company does not think in terms of absolute token performance.

“The metric that matters is your total token throughput per megawatt,” Buck said.

Buck described software that manages the power of each rack in real time and coordinates with the data center management system. On one model he cited, GPU power ranged from 600 watts to 1,900 watts. Operators provision racks for that peak. Buck said the gap between provisioned power and actual draw is capacity operators are not using. Managing to the actual draw lets them install more racks inside the same provisioned power. He put the gain at up to 40% more compute in the same megawatts.

The Energy Wall

Menard described the constraint from the operator side. Crusoe sources energy, builds data centers, manages GPUs and serves tokens. He said the company optimizes for the value it extracts from each megawatt.

“If you look at 2028, what you’re optimizing is availability of megawatts,” he said. “So, you need to be ready for that because the energy wall is coming and it’s gonna take a lot of time to remove it.”

Crusoe may offer its customers infrequent service degradations to avoid stressing the grid, Menard said, because the company shares that energy with the broader community.

Katti described a similar approach to the grid at the 8-gigawatt Ohio campus OpenAI is building with NVIDIA. Jobs can ramp down when grid demand spikes and ramp up when it drops.

“This can help stabilize the grid in Ohio,” Katti said.

Buck said a scheduler that knows where each job runs and how much power it draws can balance delivery across rows of the Ohio site. He put the gain at 30% to 40% more GPUs inside the same allocation. The site’s power system was designed for that control, from the compute trays up to the transformer.  

More From What Is Already Built

Katti also described a second source of efficiency: a model tuning its own inference. OpenAI took a checkpoint of its Astra model, already optimized for Blackwell, and ran it on Rubin with minor changes. Katti said it ran three times faster out of the box. The team then had the model optimize its own inference serving on Rubin. Over 72 hours, Katti said, the model produced another 2x. He said that work would have taken weeks or months in the past.

Two announcements applied the premise of more tokens per megawatt to installed infrastructure. Axiado previewed a platform efficiency controller that runs AI agents on management silicon. The agents scale processor voltage and frequency and drive cooling from measured conditions rather than worst-case assumptions. Hammerhead AI announced a distribution agreement with TD SYNNEX for a 1-megawatt SKU of its power orchestration software. The offering targets operators facing utility interconnection queues that run several years in major U.S. markets.

Nick Harris, founder and CEO of Lightmatter, described a different route to more capacity without new power. Optical interconnect spans a kilometer, so operators can spread current-generation GPUs across existing facilities rather than concentrating them in new high-density racks.

“It’s reusing data centers that are already there, because optics doesn’t care,” Harris said. “A kilometer reach is not a problem.”

The Water Question

Robert Hormuth, corporate vice president of architecture and strategy at AMD, addressed the public reaction to data center siting, power usage and water usage. He said the industry needs to do a better job governing all three.

“I don’t know if I want one in my backyard either,” Hormuth said.

Airsys, a global provider of specialized thermal management and engineered cooling solutions, won TechArena’s inaugural Ad Astra: AI Infrastructure Competition for LiquidRack ahead of the summit. LiquidRack integrates cooling into the rack rather than relying on facility infrastructure. The system uses patented spray cooling, which Airsys says transfers heat up to three times more efficiently than conventional methods. For data centers running LiquidRack, Airsys reports a PUE below 1.02, no water consumption and 80% less dielectric fluid than immersion cooling. Jeff Moore, VP of strategic partnerships at Aegis Cooling, an Airsys company, joined the Day 1 panel on power and cooling at scale.

The TechArena Take

The slowdown conversation and the infrastructure conversation ran on separate tracks in Santa Clara, and operators should keep them separate. Frontier labs pacing capability gains does not reduce the compute required to serve models already in production. The safety work those labs described adds to it.

What changed at this summit is the unit of account. Tokens per megawatt showed up in NVIDIA’s keynote, in Crusoe’s operating model and in the claims vendors brought to the floor. Operators plan around the power they can get. Rack-level power management, grid-aware scheduling, models tuning their own inference and cooling that uses no water all add capacity without adding megawatts.

Crusoe's Erwan Menard gave the timeline worth watching. Software moves in days, he said, and the energy world moves in multiple years. Operators planning for 2028 are not optimizing for chips. They are optimizing for the megawatts they can secure.

The best per-accelerator server result on MLPerf’s DeepSeek R1 reasoning test improved 5.7x in a year, according to MLPerf Inference v6.1 results released Wednesday by MLCommons, the benchmark setter for the AI industry.

The round set participation records, with 30 organizations submitting 486 data center and edge performance results. Five organizations and one individual contributor submitted for the first time. MLCommons listed five new processors, accelerators and platforms in the results, among them NVIDIA’s Rubin GPU and the Vera Rubin NVL72 rack-scale platform in preview. MLCommons added two tests, one for end-to-end retrieval-augmented generation (RAG) and one for agentic inference at the edge.

Round-Over-Round Gains

The DeepSeek R1 result sits at the top of a 2.7x to 5.7x range the consortium reported for per-accelerator gains over the past year. The best per-accelerator server result on the visual language model test improved 2.99x against v6.0 six months ago. Llama 2 70B, in its sixth round, showed a median per-accelerator gain of 5.58x since v4.0.

MLCommons attributed the range to software and new platforms, and it named next-generation accelerators and lower-precision formats as the two key drivers behind the Llama 2 70B gain. Miro Hodak, MLPerf Inference working group co-chair, said in a press briefing that early rounds ran FP8, while almost all v6.1 results used FP4.

Submitters also shifted which models they ran. GPT-OSS-120B drew 112 submissions, the most of any workload in the suite, unseating Llama 2 70B from the top spot. Version 6.1 also opens speculative decoding to GPT-OSS in the interactive scenario, where it joins DeepSeek R1.

The benchmark treats power measurement as optional, and many submissions skip it, said David Kanter, founder of MLCommons and head of MLPerf, when asked during the briefing Q&A whether submitters recorded wall power during runs. Shruti Koparkar, an NVIDIA product marketing senior manager, added that her company submitted no measured power this round, and that normalizing against a rack’s rated power falls short “because you are not actually drawing all the power on every workload.”

Systems Behind the Numbers

Systems got bigger and more varied. Multi-node submissions reached 16, up from 13 in v6.0. Crusoe submitted a 512-accelerator run on AMD Instinct MI355X GPUs, the largest system ever entered in the benchmark. Cisco unified NVIDIA H200 and AMD Instinct MI350X GPUs into one inference pool, which Hodak said was the benchmark’s first submission to combine accelerators from the two vendors. Cisco reported 99.9% aggregation efficiency on Llama 2 70B. MangoBoost reported that its LLMBoost software knit 32 GPUs at four sites into a single serving pool across an intercontinental network, achieving 97% scaling efficiency in offline mode.

Hodak tied distributed inference to production practice. “Distributing models across multiple nodes is a trend in the industry, especially if you want to minimize time to first token,” he said. Cloud companies running more than one vendor’s silicon need a way to ship workload between them, he said, and geographic distribution answers a need for resilience.

The new silicon MLCommons listed runs from the desktop to the data center rack. AMD’s Ryzen AI Max+ 395 is a unified-memory APU for local AI, and Intel’s Arc Pro B70 a 32GB workstation GPU for large-model inference. AMD’s Instinct MI350P brings a PCIe option for air-cooled servers. NVIDIA entered its Rubin GPU and Vera Rubin NVL72 platform in preview, the category for hardware due to ship by the next round.

The Benchmark’s New Tests

Both new tests measure multi-step workloads rather than a single pass:

End-to-End RAG measures a full question-answering pipeline rather than a single model call. Four models handle embedding, reranking, document grading and answer generation, and a sufficiency checker sends the query back around the loop up to five times. It reports documents per second for building the vector database and tasks per second for answering, where one question runs a dozen or more model calls.

Edge Agentic Inference replays 20 recorded agent trajectories drawn from real GitHub issues in SWE-bench Verified, 1,007 turns in total, against Qwen3.6-27B at a 32K context window, and reports mean single-stream latency per turn. Accuracy is gated on a separate run against the Berkeley Function-Calling Leaderboard v4.

Ramesh Chukka, who co-chairs the End-to-End RAG task force, said the effort began more than a year ago with member feedback asking for a benchmark that “measures the performance and accuracy from end to end, not just for the inference.”

Atlas Inference, a first-time submitter, ran the edge test on an NVIDIA DGX Spark and on AMD Strix Halo. “For the past few years, serious agentic work meant a datacenter round trip,” the company wrote in its submitter statement. “That assumption is what this submission is meant to retire.” The company reported that it completed the full 1,007 turns on the DGX Spark in under 64 minutes.

Lambda ran the same agentic test on data center silicon, submitting in the edge category, open division, on eight NVIDIA B200 GPUs. Chuan Li, chief scientific officer at Lambda, said the company swapped the benchmark's 27 billion-parameter reference model for Kimi K2.6, a trillion-parameter model from Moonshot AI. Li said Lambda also processed about 9% more output tokens per second on GPT-OSS-120B than it did six months ago, on the same NVIDIA GB300 hardware.

Orrick Industries, another first-time submitter, reported 274,927 tokens per second on Llama 3.1 8B from six NVIDIA B200 GPUs in the open division, where results are not required to be mathematically equivalent to the reference model.

Edge Agentic Inference already runs through MLPerf’s API-centric harness, and the RAG test is set to follow. Sixteen of the 30 submitting organizations used the harness this round, up from a single open-division submitter in v6.0. That harness is the foundation of MLPerf Endpoints, a suite that plots each system’s tradeoff between total throughput and speed per user. Endpoints opens rolling submissions in October 2026 and is slated to replace MLPerf Inference for the data center in 2027. The TechArena Take

Version 6.1 brings the suite closer to how AI gets deployed. Since it began in 2019, MLPerf Inference has measured single-shot workloads, which still cover a large share of production work: image recognition, transcription and single-pass answers. RAG pipelines and agentic coding sessions bring two more shapes into the suite. Buyers sizing infrastructure for those workloads now have peer-reviewed numbers to back up vendor assertions.

The range of systems across the round is the other story. One submission ran 512 accelerators. Another ran on a single SoC, with client and server both on the device. Cisco pooled GPUs from two vendors, and MangoBoost’s software held four sites together across an intercontinental network. A benchmark that puts all of that through a robust peer review process serves an industry buying at every one of those scales.

MLCommons is rebuilding its inference benchmark while participation sits at an all-time high. MLPerf Endpoints arrives in 2027, and Kanter said MLCommons is revisiting normalization for future inference benchmarks, calling it “absolutely critical.”