Ayar Labs on Taking AI Scale-Up Beyond the Rack

Ahead of the AI Infra Summit this September, TechArena is spending the summer with companies up and down the AI stack to learn the latest on AI infrastructure requirements. Our conversations run from hyperscalers to storage, memory and networking, out to power, cooling, and the connectivity now holding these systems together.

I had the pleasure of chatting with Vishal Chandrasekar, head of product management at Ayar Labs, a co-packaged optics pioneer, to talk about what happens when AI scale-up outgrows the single rack. We covered why copper runs out of room, what an optical fabric has to get right, and how moving light into the package changes the economics of an AI data center. Here's what I learned.

Q1. AI scale-up is pushing systems beyond the boundaries of a single rack. What are chip and system designers asking for today that they did not need before the rapid growth of AI?

A: The fundamental requirement has changed. Chip and system designers are no longer optimizing only for the performance of an individual GPU, ASIC or switch. They are trying to connect thousands of accelerators so they can operate as a single unified system, with the bandwidth and latency needed to support increasingly large AI models and inference workloads.

That creates new demands around bandwidth density, power efficiency, reach and architectural flexibility. System designers need connectivity that can extend across multiple racks without the power penalties and signal-integrity challenges associated with driving high-speed electrical signals over longer distances. They also need a solution that fits into the standard fab, packaging flows and system-in-package architectures they already use.

This is where co-packaged optics becomes essential. Ayar Labs’ TeraPHY optical engine brings optical connectivity directly into the compute or switch package, while the SuperNova external light source provides the reliability and serviceability required for data center deployment. Together, they give designers a practical path to scale beyond the rack without rebuilding their design and manufacturing flows around a proprietary architecture.

Q2. Copper has supported generations of computing infrastructure. Where does it begin to constrain AI scale-up, and why can’t the industry simply continue improving electrical connectivity?

A: Copper remains effective over short distances, but its limitations become much more pronounced as bandwidth and scale-up domain sizes increase. Moving extremely high-bandwidth electrical signals requires more power as the distance grows, while signal integrity becomes increasingly difficult to maintain. That effectively traps high-performance AI scale-up architectures within individual racks.

This creates a difficult tradeoff. System designers can keep accelerators physically close together, concentrating enormous amounts of power and heat in increasingly dense racks, or distribute compute across multiple racks and accept a connectivity penalty. Neither is sustainable as AI systems grow.

CPO removes that tradeoff by using light to extend high-bandwidth, low-latency connectivity across tens of meters. Ayar Labs’ solution delivers up to 10 times higher bandwidth, 10 times lower latency and three to five times greater power efficiency than conventional copper and pluggable alternatives. That allows compute resources to be distributed across multiple racks while continuing to function as one scale-up domain.

Q3. Ayar Labs talks about enabling thousands of GPUs across racks to operate as a single unified system. What has to be true for an optical fabric to deliver that experience?

A: The optical fabric cannot become a new bottleneck. It has to deliver extremely high bandwidth and consistently low latency across the entire scale-up domain so accelerators can exchange data as though they were part of one tightly integrated system. It also has to consume very low power so as to not alter the economics of AI deployments which are often power constrained.

It also has to provide enough bandwidth density within the package. The TeraPHY optical engine uses microring modulator technology to deliver two terabits per second (Tbps) of shoreline bandwidth density per millimeter. That makes it possible to bring significant connectivity directly into an XPU or switch package without consuming the power and package area associated with traditional electrical interfaces.

Just as importantly, the architecture has to remain flexible. AI scale-up infrastructure is being deployed using UALink, PCIe, NVLink, ESUN and other standards and specifications. Ayar Labs has designed a protocol-agnostic architecture that can support evolving requirements without locking customers into one protocol, light-source provider or vendor-specific infrastructure or manufacturing flow. That combination of performance and compatibility is what turns optics from a component-level improvement into the foundation for a scalable AI fabric.

Q4. CPO has been discussed for years, but demonstrations are very different from high-volume deployment. What separates a promising optical technology from a manufacturing-ready solution?

A: The real test is whether the technology can be manufactured, integrated and serviced using the processes the semiconductor and data center industries already rely on. A successful demonstration matters, but customers also need confidence in reliability, supply, packaging, fiber attachment, thermal performance and production yield.

Ayar Labs has developed and shipped multiple generations of the TeraPHY optical engine, with thousands of units in customers’ hands. TeraPHY is built on proven silicon photonics and standard CMOS manufacturing, using a chiplet architecture and UCIe-based electrical interface designed to fit within existing ASIC and system-in-package designs.

Manufacturing readiness also extends beyond the optical engine itself. Ayar Labs has established partnerships across foundries, advanced packaging providers, OSATs, ASIC design services and ODMs. The company has developed reference designs and manufacturing flows at the package, board and rack levels, that support high-volume assembly and field serviceability.

CPO reaches volume when it becomes a natural extension of existing semiconductor design and manufacturing processes rather than a specialized architecture customers have to build around.

Q5. As scale-up domains grow to thousands of GPUs, how does moving from copper to CPO change the economics of the compute an AI data center can deliver?

A: The economics of AI infrastructure are increasingly determined by how efficiently a system can keep expensive compute resources working productively. It is not enough to add more GPUs if the network cannot feed them data quickly enough or if the power and cooling required to connect them consume an unsustainable share of the data center’s capacity.

Copper’s limited reach forces operators to concentrate compute within very dense racks or sacrifice bandwidth as they distribute it. CPO allows compute to be spread across multiple 100-kilowatt racks while preserving the high-bandwidth, low-latency connections needed for AI scale-up. That reduces thermal concentration and gives data center architects more flexibility in how they deploy power, cooling and compute capacity.

At the same time, Ayar Labs’ scale-up CPO solution provides five to 10 times greater bandwidth and three to five times greater power efficiency than conventional alternatives. As token volumes grow, those improvements can translate into more useful compute from the same power envelope and at a lower cost per token.