VAST Data on the Data Layer Behind Every Good Token

TechArena has spent the summer connecting with companies building AI infrastructure to learn more about the challenges and evolving requirements they face.

For our latest conversation in this series, we had the opportunity to chat with Andy Pernsteiner, Field CTO at VAST Data, a company built on the idea that the data platform matters as much as the compute it feeds. That idea carries new weight now that agentic AI is turning retrieval into a real-time workload.

We talked about where bottlenecks hide along the training and inference pipeline, from WAN ingest and bursty checkpoint writes to the KV cache reads that come with every generated token; what agentic retrieval requires from the storage underneath it; and what changes when storage, database, and AI compute run as one platform instead of a chain of stitched-together systems. Here's what we learned.

Q1. VAST is built on the idea that the data platform, not just the GPU, sets the ceiling on AI performance. Why does data decide how much useful work a GPU actually does?

A: For model builders, GPUs are the most expensive resource in their pipelines. For nearly everyone else, GPUs are the scarcest resource they have, outside of skilled staff. That's why feeding them the right data matters so much.

“Right data” means a few things at once:

Prepped and clean. Data that hasn't been properly formatted, deduplicated, and cleansed causes the GPU to waste cycles processing input it can't use, or producing output that has to be thrown away and redone. Either way, that's expensive compute spent twice.
Governed and trustworthy. Data that isn't correctly tagged, secured, and governed to an organization's (or the law's) requirements can produce work that has to be discarded outright, regardless of whether the model output itself was correct. Same waste, different root cause.
Available on demand. GPUs are only earning their keep when they're actively computing a model or an inference result. Every cycle spent waiting on I/O is a cycle that's gone for good, you can't bank idle GPU time and use it later.

Q2. Training and inference move data through a long pipeline, from ingest and preparation to checkpointing and retrieval. Where along that pipeline do the real bottlenecks show up?

A: There are the obvious bottlenecks, compute, network, and storage performance, and some less obvious ones.

Training

‍Ingest: Usually, a bulk load (data acquisition) or continuous scraping (web, social, sensor feeds, etc.). The bottleneck here is typically the network (usually WAN/internet) combined with latency (distance from source to destination) and how much concurrency/parallelism you can bring to bear. Most training data starts life outside the four walls of the data center, so it has to be brought in first. That “tax” doesn't need to be repaid on every subsequent training run, except for new datasets being added. ‍
Preparation: Typically bottlenecked by compute and/or a lack of parallelism. Frameworks like Dask and Spark are good at parallelizing this work, but it takes real expertise to shard efficiently across large datasets and compute pools. CPU becomes the constraint when prep servers are undersized relative to the GPU fleet they're feeding. Increasingly, GPUs themselves are being used for parts of this step, as the major distributed frameworks add support for offloading preprocessing onto whatever compute is available. ‍
Checkpointing: Most training runs now use asynchronous checkpointing, which smooths out peaks and reduces burstiness, but there still needs to be enough available bandwidth that a run doesn't stall waiting on a checkpoint write. It's important to separate async checkpoint writes from the synchronous writes that are also part of the training loop, since a flood of bursty checkpoint traffic can starve out sync flows that, by nature, block the job from proceeding.

Inference

‍Model load: Not always relevant in a busy environment where popular models are already resident, but time-to-first-token (TTFT) is directly affected by how long it takes to load a large model into GPU memory. Frontier models are large enough now that this requires real network and storage bandwidth over highly optimized, RDMA-capable fabrics. GPU high-bandwidth memory will always outrun the network feeding it, so the job is to remove as much friction as possible from that I/O path. ‍
Prefill / prompt processing: Largely GPU/compute-bound, but this phase also writes KV cache aggressively as it's computed. In environments using external KV cache (i.e., on external storage rather than GPU memory alone), that requires high bandwidth to the storage layer and a storage subsystem that can keep up, not just with the writes, but with the reads needed to verify the current state of the cache. ‍
Decode (token generation): Once prefill is done, decode produces output tokens one at a time, and every step reads back the KV cache accumulated so far. Unlike prefill, this phase is memory-bandwidth-bound, not compute-bound: the GPU spends more time waiting on data than doing new math. It gets harder as context windows grow, since KV cache size grows right along with them. It also matters in disaggregated serving architectures, where prefill and decode run on separate GPU pools and the KV cache computed during prefill has to move to the decode pool fast enough that decode isn't left waiting. Time-per-output-token ends up being a function of how quickly that cache can be read and moved, not just raw GPU FLOPs.

Q3. Retrieval, vector search, and RAG have become heavy, real-time workloads as AI moves toward agents. What does that ask of the data layer underneath?

A: Agentic workloads turn retrieval into something closer to a real-time transactional workload than a batch job, and that changes what's required of the data layer:

KV cache. As context windows grow and agents chain multiple reasoning steps together, KV cache read/write volume grows with them. The data layer needs to serve that cache with low, consistent latency, both writing it as it's generated and reading it back to check state, without becoming the thing an agent is waiting on mid-reasoning.
Vector retrieval. An agent making a decision often triggers a vector search inline, in the critical path, not as a background task. That means the data layer has to support low-latency similarity search at scale, across potentially billions of vectors.
Associated datasets. Vector search rarely stands alone, the agent usually needs the structured or unstructured source data the vector points back to (documents, tables, metadata) to actually ground its answer. Ideally, the source data (such as a document or a text chunk) resides close to or on the same system as the embeddings, not only to reduce the complexity of the system, but also to maintain tight lineage between source and derivative, especially in environments where data can change or be deleted frequently.

Q4. VAST brings storage, database, and AI compute into one platform rather than separate systems. What does that unification make possible, and where are its limits?

A:

What it makes possible

Eliminates the data movement that fragments most AI stacks today. For example, the ETL pipelines that copy data between a storage system, a vector database, a feature store, and a compute cluster just so each system can do its one job. Data is prepped, stored, and retrieved in place.
Collapses the latency in the retrieval loop described above: vector search, KV cache, and associated datasets can be served from the same platform instead of stitched together across systems with their own network hops.
Simplifies governance and security to a single policy plane. Access controls, encryption, and lineage are enforced once, at the platform level, instead of being re-implemented (and potentially re-broken) in every downstream system that touches a copy of the data.
Lets compute run where the data already lives. Embedding generation, preprocessing, or lightweight inference can happen against data in place rather than requiring another export/import cycle.
Reduces the operational burden: fewer hardware and software systems to purchase, provision, patch, monitor, scale, and reconcile with each other.

Where the limits are

Unification is a platform capability, not a substitute for organizational discipline. VAST provides the platform and policy frameworks, but domain experts and policy owners still need to be directly involved in the design of end-to-end systems to ensure efficient and consistent pipelines.

Organizations with an already-fragmented stack need a practical migration path, not a rip-and-replace. Unification pays off fastest on new workloads and gets adopted incrementally on existing ones.

It doesn't eliminate bottlenecks that live outside the platform's control, such as WAN ingest from external sources or network fabric choices upstream of the cluster.

Q5. In the end, how does the data platform shape the cost and quality of the tokens an operator can deliver?

A: Cost and quality both trace back to the same root cause: whether GPU cycles are spent on useful work.

Eliminating bottlenecks directly lowers cost per token, because idle or stalled GPU time doesn't produce any tokens at all, it's pure overhead on top of very expensive hardware. A platform that keeps GPUs fed converts a higher percentage of the compute you're paying for into actual output.

Governance and lineage shape quality and trust. Knowing what data trained or informed a given model, and being able to prove it, is what lets an operator stand behind the tokens it delivers, particularly in regulated industries or anywhere provenance matters. It also reduces the risk of hallucinated or non-compliant output making it into production.

Cataloging and labeling ensure the right data gets used in the first place, which avoids wasting compute on irrelevant, duplicate, or low-quality data, the single most common cause of both wasted GPU cycles and degraded model output.