Graid Technology: Why AI Infrastructure Needs GPU-Driven Storage

As AI inference workloads shift toward agentic AI, memory capacity increasingly determines the cost to serve them.

This summer, TechArena has been asking the companies that build AI infrastructure how their requirements are changing as deployments scale. We sat down with Randy Kreiser, Field CTO at Graid Technology, which has developed a portfolio of storage performance and resilience products. Using GPU-accelerated RAID for KV cache offloading, he said, makes time to first token (TTFT) roughly three times faster than running with no cache offload at all.

We also discussed what a storage tier has to deliver to keep pace with the GPU and where storage fits into the industry's shift toward rack-scale AI systems designed as a whole. Here's what we learned.

Q1: AI infrastructure conversations used to start and end with FLOPS, but inference has moved the constraint to memory. From your vantage point, how did memory become the gating factor for inference economics, and why can't GPU memory alone solve it?

A: Training is limited by how fast you can process data; inference is limited by how much conversational state you can keep available. Every active request carries a KV cache, the model's working memory of the conversation, and cache demand grows with context length and concurrent users, not simply with model size.

That matters especially for agentic AI. A long-running multistep agent can create substantial cache demand even when GPU compute is not the limiting resource. The constraint is not the GPU; it is the memory available to hold and reuse state.

GPU memory alone cannot solve the problem economically. HBM comes attached to an accelerator, so adding memory often means buying compute you do not need. When cache capacity is exhausted, the system evicts state and recomputes it later. GPU utilization may still look healthy, but useful work declines because the system is regenerating context it already processed.

Q2: KV cache offloading essentially turns NVMe storage into an extension of the memory hierarchy. What has to be true of the storage layer for that tiering to work at inference speed rather than becoming the next bottleneck?

The cache must be faster to retrieve than it is to recompute. If it is not, offloading simply adds latency to the inference path.

A: The threshold is higher than many teams expect. In a controlled vLLM and LMCache benchmark using a 235 billion-parameter MoE model across four NVIDIA H200 GPUs, Linux MD RAID5 increased TTFT from 29.4 seconds with no offload to 36.6 seconds. Adding protection to an insufficiently fast storage path can make inference worse, not better.

For NVMe to serve as an effective inference-memory tier, it needs:

• A direct path into GPU memory, avoiding unnecessary host-DRAM copies.

• Predictable tail latency, because inference SLAs are governed by p99 behavior, not average bandwidth.

• Sufficient capacity in a unified namespace, avoiding manual tier management.

• Resilience without a runtime performance penalty; a rebuild cannot cut production throughput in half.

Miss any one of these, and the storage tier becomes the next bottleneck. Graid Technology's innovative volume management capabilities deliver all of these.

Q3: Traditional RAID architectures were built for a CPU-centric world and burn the very cycles and PCIe lanes AI servers can't spare. What convinced Graid Technology that RAID logic belonged on the GPU, and what does that unlock in an AI data center that a hardware RAID card or software RAID can't?

A: RAID is fundamentally parallel parity math: XOR and Galois-field operations. Traditional architectures placed that work on a RAID-controller ASIC because CPUs were poorly suited to it and there was no better parallel processor in the server.

That is no longer true. Modern AI servers already contain massively parallel GPUs, while NVMe has exposed the limits of conventional RAID designs. A hardware controller can become a bandwidth ceiling as Gen5 NVMe arrays scale up. Software RAID avoids that controller bottleneck but consumes host CPU cores and PCIe resources that AI workloads need for feeding accelerators.

GPU-accelerated RAID changes the tradeoff: SupremeRAID AE uses a small portion (approximately 4% or 6 SMs) of an installed GPU rather than a dedicated RAID card, preserves host CPU resources, and enables a direct storage-to-GPU-memory path. The result is protected, high-throughput storage designed around the AI server rather than bolted onto it.

Q4: You've positioned SupremeRAID AE around KV cache offloading, citing roughly 3x improvement in TTFT. Walk us through how GPU-accelerated storage changes the inference pipeline in practice. Where do those gains actually come from?

A: Correct, we measured 3.26x faster TTFT than no offload and 4x faster than Linux MD RAID5, reducing mean TTFT from 29.4 seconds to 9.0 seconds.

The gain is not merely faster storage. It comes from avoiding unnecessary GPU computation. TTFT is heavily influenced by prefill, the work required to process the full prior context before the model can generate its first token. In long-context agentic workflows, that prefill work is expensive and often repeated.

When the KV cache already exists, the system can retrieve it instead of recomputing it. A sufficiently fast storage read is far cheaper than rerunning attention across a large context window. That releases GPU cycles for decoding and serving additional requests. The business case is simple: Fetch must beat recompute. If it does not, the offload tier works against you. SupremeRAID is proven to be faster than recompute.

Q5: The industry is moving from components bought off spec sheets to rack-scale systems designed as a whole. As storage becomes a design partner to the GPU rather than a peripheral, how does Graid Technology's roadmap fit into that integration story, and what does storage look like in the AI data center two years out?

A: Our Agentic AI Storage Portfolio is organized by deployment scale, KV Cache Server, KV Cache Rack and KV Cache Platform, rather than by individual SKUs. That reflects a shift in AI infrastructure design: Once the rack is the unit of deployment, storage cannot be treated as a component to integrate afterward.

KV Cache Platform aligns with NVIDIA's STX reference architecture. The roadmap includes native BlueField-4 DPU execution in H2 2026 and expanded drive-count support, allowing one SupremeRAID instance to span multiple CMX chassis and present a virtualized pool to an entire rack of STX nodes.

Over the next two years, storage will increasingly be evaluated in inference outcomes rather than raw capacity: cost per million tokens served, cache-hit rate, and TTFT. Terabytes remain necessary, but they stop being the headline metric. RAID and I/O processing will follow the available parallel compute, from GPUs today to DPUs in the next phase.