WEKA vs VAST Data vs DDN vs Pure Storage: AI storage for GPU training workloads
WEKA, VAST, DDN and Pure compared for GPU training: metadata topology, checkpoint bandwidth, audited MLPerf Storage results and NVIDIA validation scope.
Your accelerator utilization trace is the only honest report card on a training storage tier. When I pull nvidia-smi samples alongside storage-side throughput counters on a stalling job, the gap almost never sits where the procurement conversation assumed it would.
Raw sequential bandwidth is usually adequate. Metadata operation rates and checkpoint flush windows are where the idle GPU-hours accumulate.
That distinction decides which of these four platforms is correct for your workload, because they resolve different bottlenecks.
WEKA and DDN EXAScaler distribute both data and metadata across nodes.
VAST separates stateless compute from a shared media pool with a dedicated metadata tier. Pure Storage, now trading as Everpure, splits a metadata engine from independently scaled data nodes in FlashBlade//EXA.
This covers training and checkpointing only. Inference serving, KV cache tiering, and vector database workloads have different access patterns and belong in a separate evaluation.
The four I/O patterns that stall GPU training jobs
Every storage decision in this space reduces to which of four patterns dominates your job, and each one fails differently.
Sequential read throughput across the epoch loop
Start with arithmetic rather than a vendor datasheet. An H100 running a vision model at 2 GB/s of sample throughput needs 16 GB/s across an eight-GPU node, and 128 GB/s across sixteen nodes. Once your dataset exceeds aggregate host memory, the page cache stops absorbing re-reads and every epoch hits the storage tier cold.
Dataloader prefetch depth masks short latency spikes but cannot manufacture bandwidth. If your prefetch queue drains faster than the storage tier refills it, utilization sags and stays there.
Metadata operations on file-per-sample datasets
A corpus of 40 million 12 KB images is a metadata workload wearing a bandwidth costume. Each sample costs an open, a stat, a read, and a close, and PyTorch dataloader workers multiply that op rate by the worker count per node.
I have seen a cluster sitting at 3 GB/s of measured throughput while the file system serviced 400,000 metadata operations per second and every GPU sat at 30% utilization. Adding bandwidth to that system would have changed nothing.
Checkpoint burst writes
Checkpointing inverts the pattern completely. All ranks write simultaneously, the write is synchronous to job progress, and every accelerator idles for the full flush duration.
Checkpoint payload size is a function of parameter count and optimizer state layout, and it does not shrink when you add nodes. A 405B-parameter job writes the same volume on 64 GPUs as on 8,000.
Resume reads and validation passes
The pattern nobody benchmarks and everybody eventually needs. A resume after node failure reads the full checkpoint back with a cold cache, and validation passes interleave small random reads into a job tuned for streaming.
How WEKA, VAST, DDN and Pure move data to the GPU
Four architectures, four different answers to where metadata lives and how the client reaches the media.
WEKA NeuralMesh: sharded metadata across a distributed parallel file system
WEKA distributes data and metadata across every node in the cluster, so there is no dedicated metadata server to saturate. Metadata is sharded into a large number of independent units spread across the cluster, which is why the architecture scales metadata operation rate with node count rather than with a controller pair.
The client is a native POSIX driver running in user space that bypasses the kernel page cache and talks NVMe over fabrics to the storage nodes. That design removes kernel overhead from the data path at the cost of a client component you have to install, version, and manage on every GPU node.
NeuralMesh tiers to object storage behind the namespace, and NeuralMesh Axon pools NVMe local to GPU servers into the same namespace. WEKA states WEKApod 3 reaches 10.2 TB/s of throughput and 210 million IOPS per rack, which is a vendor figure with no audited submission behind it.
VAST Data DASE: stateless compute over shared NVMe with an SCM metadata layer
VAST separates the two halves of a storage system. Stateless containers called CNodes hold no persistent state and front a shared pool of QLC NVMe enclosures over NVMe-oF, which means any CNode can serve any request without cache coherency traffic between peers.
Metadata and incoming writes land in a storage-class memory tier that every CNode addresses directly. This is the design decision that matters most: the SCM tier absorbs write bursts before they reach low-endurance QLC media, and it doubles as a globally shared metadata store.
Clients reach VAST over standard NFS with RDMA and multipath rather than a proprietary parallel client. VAST argues explicitly that a parallel file system is unnecessary for AI training, and that S3 checkpointers and data loaders have already displaced POSIX semantics in modern training stacks.
The wide erasure coding that the SCM buffer enables is genuinely unusual: VAST states it tolerates up to four SSD failures with as little as 2.7% capacity overhead.
DDN EXAScaler and Infinia: Lustre-derived striping and a key-value object path
EXAScaler descends from Lustre and carries its structural separation of metadata targets from object storage targets. Files stripe across OSTs, and stripe count and stripe size are tunable per file or per directory, which gives you real control and real opportunity to misconfigure.
Lustre's metadata architecture is the honest caveat. The design has historically concentrated metadata service in a way that creates a chokepoint under high stat rates, and Distributed Namespace addresses it by spreading directories across multiple metadata targets. Configuring DNE correctly is not automatic.
DDN's audited v2.0 submission reported 120.68 GB/s on 3D U-Net across multiple nodes, and up to 640 simulated H100 accelerators on ResNet-50. On checkpointing it reported 30.6 GB/s read and 15.3 GB/s write.
Infinia is a separate key-value object platform aimed at inference and metadata-heavy access, and it is not the training engine in DDN's stack.
The Lustre lineage also means the operational model assumes HPC-grade expertise in house. That assumption holds in national labs and breaks in enterprise IT teams, a pattern I covered in more depth in why real partnership matters in high performance computing.
Pure FlashBlade//EXA: disaggregated metadata core with independently scaled data nodes
FlashBlade//EXA uses the existing FlashBlade controller pair as a metadata core running the Purity metadata engine, then scales commodity QLC data nodes separately across an RDMA fabric. Metadata and capacity scale on independent axes, which is the structural difference from FlashBlade//S.
The v3.0 submission exposed the protocol split precisely: 30 FlashBlade//EXA blades with 120 DirectFlash Modules handling metadata, paired with 30 Linux NVMe data nodes for roughly 866 TB usable in a single file system, serving metadata over pNFS on TCP and bulk data over NFSv3 on RDMA. No proprietary client is involved.
Independent scaling has a boundary worth probing in a POC. Fixed cache ratios between the data and metadata layers can force the two to grow in step at certain configurations, which erodes the flexibility the architecture is built to provide. If you already run FlashBlade//S, the broader positioning against conventional enterprise arrays sits in our Pure Storage vs NetApp vs Dell PowerStore comparison for AI workloads.
GPUDirect Storage and the client-side data path
All four support GPUDirect Storage, so support is not a differentiator. What differs is the CPU cost per GB delivered and how the path behaves at depth.
GDS lets the storage stack DMA directly into GPU HBM through cuFile, skipping the bounce buffer in host memory that a conventional path requires. That saves a memory copy and, more usefully, frees host CPU cycles your dataloader workers need.
A native POSIX client in user space and an NFS-over-RDMA path reach GPU memory through different amounts of code, and the host CPU utilization per GB is the number to record during a POC.
Measure it alongside throughput, because a client that hits target bandwidth while consuming 40% of your host cores has taken capacity away from data augmentation.
DPU offload moves the storage client off the host CPU entirely. All four vendors have announced work on NVIDIA BlueField-4, and the practical question is whether the offload covers your protocol on the DGX generation you are buying.
What MLPerf Storage measures and how to read the results
MLPerf Storage is the only audited benchmark in this category, and it reports the right metric: how many simulated accelerators a system keeps above 90% utilization.
What changed in v3.0
MLCommons published v3.0 results on 1 September 2026, and the round restructured the suite. Training now uses Unet3D and RetinaNet with accelerators simulated as B200 or MI355 rather than H100.
Checkpointing became a first-class workload across 8B, 70B, 405B and 1.25T-parameter Llama 3-style models, measured on write and read duration. Vector database and KV cache workloads were added, along with an S3 access layer alongside POSIX.
Nineteen organizations submitted 143 results, eleven of them first-time submitters including Azure, NVIDIA, Nebius and Everpure.
Why v2.0 and v3.0 numbers cannot be compared
Moving simulated accelerators from H100 to B200 raises the per-accelerator bandwidth bar, so a v3.0 accelerator count and a v2.0 accelerator count measure different things. Any table that ranks the two rounds together is wrong. This is the most common misreading of these results.
Submission coverage across rounds
The audited record and the deployment record point at different vendor lists, and that is a fact about evidence availability rather than a performance claim.
Two verification notes. StorageReview's node-scaling summary lists 833.0 GiB/s read at 30 data nodes while the 1.25T-parameter run at 1,024 accelerators is reported at 588.28 GiB/s, so confirm which parameter count you are reading before you compare.
And WEKA's absence from v2.0 should be checked against the MLCommons results pages directly, since round-by-round participation is the kind of detail secondary coverage gets wrong.
Checkpoint write bandwidth and model parameter scale
Checkpoint payload is arithmetic you can do before you talk to any vendor. In mixed precision, bf16 weights cost 2 bytes per parameter, and Adam optimizer state commonly adds an fp32 master copy plus two moment tensors for roughly 12 bytes per parameter.
A 70B-parameter job therefore writes about 140 GB of weights and 840 GB of optimizer state, close to 1 TB per checkpoint. Verify the layout against your framework, because sharded optimizer implementations change the per-rank distribution even when the aggregate holds.
The v3.0 numbers corroborate this arithmetic. Everpure's audited 877.52 GiB/s sustained over 17.74 seconds implies roughly 15.2 TiB written, which works out near 12.4 bytes per parameter at 1.25T parameters, assuming that duration covers a single checkpoint write.
Note that the read side came in slower than the write side in that submission, at 588.28 GiB/s over 28.99 seconds. Resume is the operation you perform under pressure after a node failure, so a read path slower than the write path directly extends your recovery window.
Checkpoint interval math then decides how much of this matters. At 8,000 accelerators with a mean time between failures measured in hours, you checkpoint frequently, and a 20-second flush every 10 minutes costs roughly 3% of your total GPU-hours. That number is your budget ceiling for the storage tier.
Metadata throughput on small-file training datasets
Test metadata against your actual file-size distribution rather than a synthetic mix. A corpus with a median file size of 8 KB and a long tail to 4 MB behaves nothing like a uniform 1 MB test set.
Distributed metadata across all nodes, a shared SCM tier, and a dedicated metadata engine each scale differently. The sharded and shared-tier designs add metadata capability as the cluster grows, while a controller-pair engine adds it in discrete steps when you add controllers.
Before you buy anything, check whether the dataloader can move the bottleneck for free. Packing a file-per-sample corpus into WebDataset shards or TFRecord files converts a metadata workload into a streaming workload and can eliminate the problem entirely. I have watched teams specify a metadata-optimized tier for a corpus that should have been sharded in preprocessing.
Fabric and media constraints behind the published numbers
Published throughput assumes a fabric that most enterprise networks do not have. Work backward from your accelerator count to required aggregate bandwidth, then to port count and fabric class, and check the oversubscription ratio on the leaf-spine path between GPU nodes and storage.
InfiniBand and RoCE both carry the storage path, and the choice interacts with what your GPU fabric already runs. Splitting storage onto Ethernet while training traffic runs on InfiniBand means two fabrics to operate, which has consequences covered in our network infrastructure vendor comparison.
QLC media raises a specific question for checkpoint-heavy jobs. QLC delivers capacity at lower cost per terabyte with lower write endurance, and a job checkpointing 1 TB every ten minutes writes 144 TB per day.
Run that against the drive endurance rating and the write amplification the buffer tier introduces, then ask the vendor how the buffer protects the media.
Power efficiency is now measurable independently. MLCommons reported on-premises checkpoint-write submissions at a median of 14 GB/s per watt with a maximum of 201, and Unet3D read submissions at a median of 34 GB/s per watt with a maximum of 277.
Those medians are a useful sanity check against a vendor efficiency claim, and they connect to the density constraints I worked through in AI data center strategy for power and capacity.
NVIDIA validation scope and what it actually certifies
All four hold DGX SuperPOD validation, which makes the certification a scope question rather than a yes-or-no gate. DDN is a SuperPOD reference platform, WEKA has a WEKApod reference architecture, VAST is validated, and Pure's FlashBlade//S carries certification for DGX B300 SuperPOD.
Confirm three things in the current reference architecture before you commit: the exact product SKU that was validated, the DGX generation it was validated against, and the fabric topology used in the validated configuration. A certification against an older DGX generation on a fabric you are not deploying tells you very little about your build.
Failure modes and operational characteristics
Each architecture carries a specific operational cost, and these are the ones that surface after deployment.
Matching platform architecture to your workload profile
Apply the gates before comparing any performance figure, because a platform that fails a gate is out regardless of how it benchmarks.
Gate one. Validation scope covers your DGX generation, your SKU, and your fabric topology.
Gate two. Metadata topology matches your dataset file-size distribution, tested at your median and tail.
Gate three. In-house operational expertise matches what the platform assumes. No Lustre skills and no appetite to acquire them rules out self-managed EXAScaler.
Gate four. Your existing or budgeted fabric class matches what the published figures required.
Past the gates, segment by dominant pattern. Checkpoint-dominated jobs at large parameter counts point toward platforms with audited checkpoint data at your scale.
Metadata-dominated small-file corpora point toward distributed or shared-tier metadata, or toward fixing the dataset format first. Streaming-dominated jobs on sharded datasets are the easiest case and the one where existing enterprise flash sometimes suffices, a question worth reading alongside how AI is forcing a rethink of cloud storage and data fabrics.
If your GPUs are rented rather than owned, the shortlist narrows to what your provider offers, and the architecture comparison still applies to whichever options are available in that environment.
How to test this before you commit
Six tests, run against your data rather than a vendor corpus.
- Metadata ops/sec at your real file-size distribution, with dataloader worker counts matching production.
- Checkpoint burst write at your parameter count, with all ranks writing concurrently, recording flush duration rather than peak bandwidth.
- Cold-cache resume read of a full checkpoint, timed end to end.
- GDS throughput to your GPU generation, with host CPU utilization recorded in parallel.
- Sustained run through a node failure, measuring rebuild impact on job throughput.
- Expansion under load, adding capacity while a job runs.
Record accelerator utilization on every test. It is the only number that tells you whether the storage tier is actually the constraint.
The instrumentation point is the one worth keeping. Sample GPU utilization and storage-side counters together, identify which of the four patterns is producing your idle accelerator time, and let that determine the architecture. A tier specified against the wrong pattern will meet its throughput spec and leave your utilization exactly where it started.
Comparing AI storage vendors for a training build?
TechnologyMatch connects you with pre-vetted AI infrastructure and storage vendors that match your accelerator count, dataset profile, and fabric constraints. No cost to buyers, no vendor pushed at you, and no obligation to talk to anyone you don't shortlist yourself.
FAQ
Is WEKA or VAST Data better for GPU training workloads?
Neither is better across the board, because they resolve different bottlenecks. WEKA shards metadata across every node in the cluster and uses a native user-space POSIX client, which suits metadata-heavy corpora of many small files. VAST separates stateless compute nodes from a shared QLC pool with a storage-class memory tier and serves clients over standard NFS with RDMA, which removes the proprietary client from your GPU nodes. Match the choice to whether metadata operations or streaming bandwidth dominates your job.
Do you need a parallel file system for AI training?
Not always, and the industry disagrees on this point. WEKA and DDN EXAScaler are parallel file systems that distribute data and metadata across nodes for concurrent client access. VAST argues explicitly that a parallel file system is unnecessary for AI training, pointing to the adoption of S3 checkpointers and data loaders in modern training stacks. If your dataset is packed into large sequential shards, a well-provisioned NFS-over-RDMA tier can keep accelerators fed without parallel client semantics.
Why do GPUs sit idle during AI training?
Four distinct I/O patterns cause idle accelerator time, and they fail differently. Epoch streaming reads stall when aggregate bandwidth falls short of per-accelerator demand. Small-file datasets stall on metadata operation rates rather than bandwidth, so utilization can sit at 30% while throughput looks modest. Checkpoint writes drop utilization to zero for the full flush duration. Resume reads after a node failure extend recovery with a cold cache. Sample GPU utilization and storage counters together to identify which pattern is active.
What is MLPerf Storage and why does it matter for AI storage selection?
MLPerf Storage is the only independently audited benchmark for AI storage systems, published by MLCommons. It reports how many simulated accelerators a storage system keeps above 90% utilization, which measures the actual problem rather than peak throughput. Version 3.0, released on 1 September 2026, covers training with Unet3D and RetinaNet, checkpointing across 8B to 1.25T-parameter models, plus vector database and KV cache workloads. Nineteen organizations submitted 143 results in that round.
Can you compare MLPerf Storage v2.0 and v3.0 results directly?
No. Version 3.0 moved its simulated accelerators from H100 to B200 and MI355, which raises the per-accelerator bandwidth requirement, so an accelerator count from v3.0 and one from v2.0 measure different things. Any table that ranks results across the two rounds together is misleading. Compare within a single round only, and check which parameter count and configuration each figure came from before drawing a conclusion.
Which AI storage vendors submitted audited MLPerf Storage results?
Coverage is uneven across rounds and worth checking before you accept any performance claim. DDN submitted to v1.0 and v2.0 but was absent from v3.0. WEKA submitted to v1.0 with no checkpointing data in the current round. VAST Data has not submitted to any round. Pure Storage, now trading as Everpure, made its first submission in v3.0. Verify current participation against the MLCommons results pages, since vendors skip rounds for scheduling reasons and absence is not evidence of poor performance.
How much storage bandwidth does each GPU need for training?
Work backward from your sample throughput rather than from a vendor datasheet. An H100 consuming samples at 2 GB/s needs 16 GB/s across an eight-GPU node and 128 GB/s across sixteen nodes. Translate that aggregate into port count and fabric class, then check the oversubscription ratio on the leaf-spine path between GPU nodes and the storage tier. Published vendor throughput assumes a fabric most enterprise networks do not have in place.
How large is a model checkpoint and how often should you write one?
Checkpoint size follows parameter count and optimizer state layout. In mixed precision, bf16 weights cost roughly 2 bytes per parameter while Adam optimizer state commonly adds about 12 bytes per parameter for an fp32 master copy and two moment tensors. A 70B-parameter job therefore writes close to 1 TB per checkpoint. Interval depends on mean time between failures at your scale, and the payload does not shrink as you add nodes, so a 405B job writes the same volume on 64 GPUs as on 8,000.
Does DDN EXAScaler require Lustre expertise to operate?
Yes, and this is the main operational constraint on the platform. EXAScaler descends from Lustre and inherits its separation of metadata targets from object storage targets, with stripe count and stripe size tunable per file or directory. Distributed Namespace configuration for spreading directories across multiple metadata targets is a manual exercise. The operational model assumes HPC-grade skills in house, which holds in research computing environments and often breaks in enterprise IT teams.
Do WEKA, VAST, DDN and Pure Storage all support NVIDIA DGX SuperPOD?
All four hold DGX SuperPOD validation, so certification is a scope question rather than a pass-or-fail gate. Confirm three details in the current NVIDIA reference architecture before committing: the exact product SKU that was validated, the DGX generation it was validated against, and the fabric topology used in that configuration. Validation against an older DGX generation on a fabric you are not deploying tells you very little about your build. All four also support GPUDirect Storage, so GDS support alone does not differentiate them.
