
AI infrastructure providers have spent years preparing for the growth of inference. According to Dave Driggers, founder and CEO of Cirrascale, inference workloads have surged from 20% to 60–70% of deployments in just 12 months. His experience offers a firsthand look at the technical and economic challenges involved.
Cirrascale often gets grouped with the neoclouds, but the company has nearly three decades of experience in high-performance computing (HPC), supporting multiple waves of AI adoption. Dave calls it a “30-year-old startup,” which has supplied infrastructure for voice recognition applications associated with Siri, Google, and Swype, followed by autonomous driving development with Tesla, Zoox, Uber, and Lyft. Generative AI represents the third wave, with inference now accounting for most of its new deployments.
Inference Changes the Hardware Lifecycle
A year ago, Cirrascale’s workload mix was approximately 70 to 80% training and 20% inference. Today, inference accounts for 60 to 70% of the mix.
The growing demand for inference affects hardware requirements. Models range from roughly 1 billion to several hundred billion parameters, with varying performance and memory demands. As model sizes increase, training often requires newer accelerators. Cirrascale typically repurposes training equipment for inference after approximately two years.
Dedicated inference systems, meanwhile, may remain in service for four years or longer. For infrastructure providers, those longer lifecycles make hardware selection and total cost of ownership (TCO) increasingly important.
Real-time and near-real-time workloads frequently experience lower utilization outside business hours. GPU utilization for one Cirrascale customer drops 30–40% during nights and weekends.
To improve efficiency and reduce costs, operators increasingly use that available capacity for fine-tuning and reinforcement learning. Although inference requires relatively little communication between GPUs, training-related workloads need high-bandwidth, low-latency interconnects. As a result, providers may still need to deploy inference clusters with tightly coupled networking, such as InfiniBand, to enable both workload types on the same infrastructure.
When I asked Dave about the biggest infrastructure constraint, his answer was immediate: data center availability.
Cirrascale tackles constrained data center capacity with its build-to-order approach rather than deploying infrastructure speculatively. The company works with customers to understand their models and performance requirements.
Dave emphasized the importance of maximizing available power and cooling.
“I don’t want to take a 10-megawatt data center and only get eight megawatts out of it,” he explained.
A year ago, nearly all of Cirrascale’s deployments used InfiniBand. Today, that proportion has dropped to approximately 50 to 60%. For inference-focused clusters, Dave increasingly favors Ethernet, which simplifies management of multiple smaller workloads while supporting intermittent fine-tuning. Providers also face rising storage costs as they retain larger datasets for reinforcement learning.
Cirrascale traditionally relied on two storage tiers: high-performance parallel flash, including WEKA, for training and lower-cost object storage for other data. Dave reported, however, that high-performance storage costs have increased four-to-fivefold over the past year, prompting the company to introduce an intermediate tier that balances performance and cost.
Dave identified unpredictable costs as one of the two major barriers to enterprise AI adoption, alongside data security and privacy. He recommends starting with a single use case, establishing its performance and cost profile, and expanding once the economics are understood. Cirrascale is also developing tools to help customers measure and manage consumption within private environments.
Cirrascale now focuses on enterprises moving inference from pilot projects into production, rather than primarily on organizations building their first training clusters.
At that stage, cost per token becomes a critical metric: Dave noted that even a 10% reduction in inference costs can determine whether a deployment moves forward.
Cirrascale supports accelerators from AMD, NVIDIA, Groq, and SambaNova, evaluating TCO at the model level to identify configurations that meet service-level agreements (SLAs) at the lowest practical cost.
What stood out to me from my conversation with Dave is how quickly inference economics have become central to AI infrastructure decisions.
Providers need to maximize power utilization, accommodate changing workload patterns, control storage expenses, and give enterprise customers predictable operating costs. Hardware performance remains essential, but deployment efficiency increasingly determines whether AI projects can scale economically.
With inference now the dominant workload, the ability to optimize infrastructure for specific models and business requirements will be critical.
Watch my full conversation with Dave Driggers on “In the Arena” to learn more about Cirrascale’s approach to production AI infrastructure.