How Solidigm Unlocks GPU Memory Walls for AI

Ahead of AI Infra Summit, TechArena has been talking to companies building the AI data centers of tomorrow. We caught up with Jeniece Wnorowski, Director of Content Strategy & Industry Expert Programs at Solidigm, a company built on decades of memory and storage expertise now focused squarely on keeping GPUs fed as AI workloads scale. As models grow larger and context windows stretch longer, the gap between compute and the data it depends on is becoming one of the sharpest constraints in AI infrastructure.

We talked about why storage requirements have shifted from cheap, reliable capacity to keeping the GPU from ever waiting; where the data path most often breaks down, and what it really takes to size storage for the bursty, synchronized write demands of checkpointing at scale.

Q1: What are AI builders asking storage to do today that they weren't before the AI boom?

A: The shift is from "store it cheaply and reliably" to "keep the GPU from ever waiting." Four demands stand out.

First, raw density per rack unit: AI clusters are power- and space-constrained before they're capacity-constrained, so builders want maximum terabytes per watt and per slot. AI ambitions are grounded by the realities of physical footprint and power constraints, and high-capacity SSDs can dramatically reduce the storage footprint compared with HDD-based solutions. Second, sustained high-throughput reads at massive scale for training data ingestion and checkpoint restores, not just peak IOPS. Third, predictable latency at scale. AI pipelines need storage that can sustain throughput without long-tail latency spikes that leave expensive accelerators idle. And fourth, storage that sits physically and logically closer to the GPU itself.

That last point reflects a fundamental shift in AI infrastructure. Inference, RAG, vector databases, model repositories, and feature stores all require fast access to much larger active datasets than traditional enterprise applications. As models outgrow available HBM, SSDs increasingly act as a memory-expansion tier, storing model weights or KV-cache data that can be swapped in and out of GPU memory on demand. Solidigm partner Metrum AI demonstrated how fast SSDs can extend effective GPU memory capacity by serving as a storage tier for model components not actively resident in GPU RAM

Q2: Where in the data path does feeding the GPU break down most often today?

A: The most common failure points are at the two ends of the training pipeline: data ingestion and checkpoint write/restore.

During training, if storage cannot sustain the random-read throughput needed to keep shuffled batches flowing, GPUs sit idle waiting on data loaders, which is an expensive way to burn compute budget. During checkpointing, entire jobs often pause while model state is written to persistent storage. If those writes are slow, every GPU in the cluster idles simultaneously.

A third bottleneck is emerging between memory and compute: the KV cache and the memory wall. In inference, every token of context generates key/value state that consumes GPU memory. As context lengths, batch sizes, and model sizes grow, KV cache can quickly exhaust available HBM. Operators are then forced to limit context windows, reduce batching efficiency, recompute state, or introduce a lower storage tier. When data is offloaded, storage becomes part of the inference path itself. SSD-backed memory expansion can help hold model weights or KV-cache data that are swapped into GPU memory as needed, extending model capability beyond the limits of HBM alone.

This is one reason the industry is increasingly focused on architectures that move data more directly between SSDs and GPUs, minimizing CPU and software-stack overhead.

Q3: What do teams most often get wrong about storage for checkpointing at scale?

A: The most common mistake is sizing storage around capacity and average throughput rather than the bursty, synchronized write behavior that checkpointing actually creates. Hundreds or thousands of GPUs may attempt to flush model state at nearly the same moment, creating a concentrated “write” storm that can overwhelm infrastructure designed around average utilization.

Organizations also underestimate the tradeoff between checkpoint frequency and productivity. Checkpoint too rarely, and failures can result in hours of retraining and lost compute. Checkpoint too often, and the storage layer itself becomes the bottleneck.

There's also frequently a mismatch between endurance assumptions and real-world behavior. Teams often purchase expensive high-endurance media for workloads that are primarily large, sequential writes rather than small random overwrites. Understanding the actual checkpoint profile is often more important than simply optimizing for maximum endurance specifications.

Q4: As drive capacities push past 100 terabytes, what does that density solve for AI builders, and where does it stop helping?

A: What ultra-high-capacity drives solve is largely physical and economic: fewer drives, fewer rack slots, lower power consumption, less cooling overhead dedicated to storage, and simpler deployment when a single storage platform can span multiple capacity points.

Where density stops helping is when capacity growth outpaces performance growth. Capacity does not scale linearly with IOPS or bandwidth per drive, and ultra-high-capacity SSDs are optimized primarily for density and efficiency rather than maximum performance. Beyond a certain point, adding capacity alone can simply concentrate a bottleneck.

The other consideration is operational resilience. A 122TB drive contains significantly more data than smaller-capacity devices, increasing the importance of data protection strategies, erasure coding, and rebuild planning. However, capacity and resilience should be evaluated together. Modern SSDs can restore data dramatically faster than HDD-based alternatives, meaning organizations must consider both failure impact and recovery time when evaluating the tradeoffs of higher-capacity media.

Q5: As AI scales, what should operators understand about how storage shapes the cost and quality of the tokens they deliver?

A: The core idea is that GPU utilization and model quality are the outcomes storage ultimately influences.

GPU compute is the scarce and expensive resource in modern AI infrastructure, and storage's job is to ensure that resource remains productive. Every idle GPU second caused by a stalled data loader, slow checkpointing operation, storage-network congestion, or memory bottleneck directly increases training costs and degrades economics.

Storage decisions increasingly affect model quality as well. Faster, denser storage enables organizations to train on larger and more diverse datasets, checkpoint more frequently without sacrificing throughput, and scale model size without making I/O the limiting factor.

On the inference side, the emerging role of SSDs as a memory-expansion tier is especially important. SSD-backed storage can hold model weights or KV-cache data that overflow HBM, allowing operators to support longer context windows and higher user concurrency. That directly affects both the quality of the tokens a model can generate, because it can access more context, and the cost of those tokens, because each GPU can serve more requests efficiently.