Every team building a serious AI training cluster makes the same mistake: they agonize over GPU selection for months, then cobble together a storage layer in the last two weeks before go-live. I have watched this pattern play out more times than I care to count over twenty years in cloud infrastructure. The GPUs land, the cluster comes online, and training throughput lands somewhere between disappointing and catastrophic. The GPUs are idle. Not because of the network fabric, not because of the model code, but because the file system cannot feed them fast enough.
Storage I/O is the silent killer of GPU utilization in large training clusters. The pattern is predictable: a training job loads a batch of data, each GPU waits for its portion of the dataset, the checkpoint writes at the end of every epoch take longer than expected, and the whole pipeline backs up. The more GPUs you add, the worse the problem gets, because you have more clients all hammering the same storage layer simultaneously.
This article is the guide I wish had existed when I started building HPC and AI infrastructure. We are going to cover what parallel file systems actually are and how they work, the major options (Lustre, WEKA, BeeGFS), the managed cloud offerings versus self-hosted, and how to match a storage architecture to your actual workload pattern.
Why Conventional Storage Fails AI Training Workloads
Before getting into specifics, it is worth understanding why normal storage options fall short. The three alternatives most teams reach for first are object storage (S3 or equivalent), NFS (a network file share), and local NVMe SSDs.
Object storage is cheap and scales infinitely. The problem is latency. Object storage is built around HTTP-style request/response semantics, and the per-operation overhead makes it a poor fit for workloads that need to open and read millions of small files in tight loops. For datasets stored as many small files (like ImageNet’s 1.2 million individual JPEGs), object storage becomes a bottleneck immediately. You can work around this with data format changes (WebDataset, TFDS sharding, HDF5), but you are fighting the tool.
NFS is familiar and easy to mount. The problem is that a single NFS server, even a fast one, becomes a bottleneck when you have hundreds of GPU clients all reading in parallel. NFS with a single server exports a single stream of data. You can throw more bandwidth at the server, but a single metadata path means every file open, close, stat, and read serializes through one point of coordination.
Local NVMe is fast. Very fast. An NVMe SSD can sustain several gigabytes per second of sequential read bandwidth. The problem is that local storage is not shared. In a multi-node training run, you either copy the entire dataset to every node (expensive, slow, storage-wasteful) or you accept that some nodes will run out of their cached data and stall while waiting for the network to refill their local buffer.
Parallel file systems solve all three of these problems by distributing both data and metadata across multiple servers, presenting a single POSIX namespace to all clients, and stripping files across multiple storage targets so that read bandwidth scales with the number of drives in the cluster.
How Parallel File Systems Actually Work
There are two foundational concepts in parallel file systems: striping and distributed metadata.
Striping means that a single file is split into chunks (often 1 to 4 MB each) and those chunks are spread across multiple physical storage targets. When a client reads a large file, it can pull chunks from all the storage targets simultaneously. Aggregate read bandwidth equals (roughly) the number of storage targets multiplied by the per-target bandwidth. This is why a parallel file system with 32 NVMe-backed storage servers can deliver dramatically higher throughput than any single server, no matter how well-provisioned.
Distributed metadata is the harder problem. In a traditional file system, metadata operations (open, close, mkdir, stat, readdir) go through a single service. In training workloads that access millions of files, that single metadata service becomes the ceiling on throughput. Parallel file systems solve this by sharding metadata across multiple metadata servers, typically by directory or by inode range. The challenge is consistency: every client needs to see the same directory tree. Getting this right without either adding too much coordination overhead or allowing stale reads is where the different implementations diverge significantly.
The client-side piece matters too. Most parallel file systems include a custom kernel module or FUSE driver that handles client-side caching, prefetching, and aggregating small I/O operations before they hit the network. A well-tuned client can hide significant latency by pipelining requests and maintaining a read-ahead buffer sized to the training job’s access pattern.
Lustre: The HPC Standard That Cloud Adopted
Lustre is the dominant parallel file system in HPC. It powers a substantial fraction of the world’s TOP500 supercomputers, and it has been doing so for over two decades. If you have ever run a job on a national lab cluster or a university HPC system, you almost certainly ran it on Lustre.
The Lustre architecture separates three roles: the Management Server (MGS) handles cluster configuration; Metadata Servers (MDS) handle metadata operations with their associated Metadata Targets (MDT) on disk; and Object Storage Servers (OSS) handle data I/O against Object Storage Targets (OST). A client mounts the file system and talks to the MDS for metadata and the OSS for data, often with separate network paths for each.
The strength of Lustre is its battle-hardened maturity. The failure modes are well understood, the tuning options are extensive, and the ecosystem of tools for performance analysis (lctl, lfs, Lustre jobstats) is rich. You can tune stripe width, stripe count, and stripe size per file or directory to match the access pattern of different data types.
The weakness of Lustre is operational complexity. Running Lustre yourself requires expertise. The MDS is a single point of coordination for metadata (though Lustre DNE, Distributed Namespace, allows sharding metadata across multiple MDS), and getting failover right is non-trivial. This is why managed cloud offerings have taken off.
AWS FSx for Lustre is the most mature managed Lustre offering. It provisions a fully managed Lustre file system inside your VPC, with SSD and HDD-backed options, and integrates natively with S3 for data repository linking. FSx for Lustre added an Intelligent-Tiering storage class in 2025 that provides elastic capacity scaling. SSD-backed persistent FSx for Lustre is priced per GB per month (check the current AWS pricing calculator for your region, as rates vary by storage class and throughput tier). For a 100 TB dataset staging area, the cost is a meaningful line item regardless of tier. The S3 data repository integration is genuinely useful: you can keep your canonical dataset in S3 (cheap), link it to an FSx for Lustre file system, and the file system lazily imports files on first access and exports modified files back. For training jobs that read the full dataset once per epoch, this works well.
Google Cloud Managed Lustre made a significant jump with the Next ‘26 announcements. As of April 2026, Google claims throughput up to 10 TB/s for a single instance, which is a 10x increase from the previous year, and they offer three performance tiers: 250 MBps per TiB (cost-optimized), 500 MBps per TiB (balanced), and 1,000 MBps per TiB (full performance for training). The capacity ceiling is roughly 8 PiB. Per Google Cloud’s pricing page, the us-central1 rate runs approximately $0.000822 per GiB per hour, which works out to roughly $12,000 per month for a 20 TiB instance at the 1,000 MBps tier. For large-scale AI training runs that can saturate that bandwidth, the cost-per-training-step math often still favors managed Lustre over paying for underutilized GPUs waiting on slower storage.
WEKA NeuralMesh: Distributed Everything
WEKA (formerly WekaIO) takes a fundamentally different architectural approach from traditional Lustre. Where Lustre has a distinct MDS tier for metadata and an OSS tier for data, WEKA’s NeuralMesh architecture distributes both metadata and data across every node in the cluster. There is no dedicated metadata server. There is no single coordination point.
The NeuralMesh design uses virtual metadata servers that scale dynamically as you add nodes. Every node in the WEKA cluster participates in both metadata and data services. The metadata is distributed across the cluster using a distributed hash table approach, so file operations fan out to multiple nodes for coordination but no single node is the bottleneck. WEKA claims this architecture eliminates the metadata bottleneck that plagues Lustre and BeeGFS at scale on workloads with many small files.
For AI training workloads, WEKA has one capability that stands out: GPUDirect Storage support. GPUDirect Storage is an NVIDIA technology (now also supported by AMD) that allows data to flow directly from the storage layer to GPU memory without bouncing through CPU memory. In a traditional path, data comes from disk to CPU DRAM, and then a CUDA memory copy moves it from CPU DRAM to GPU memory. With GPUDirect Storage, the DMA transfer goes directly from the storage subsystem into GPU BAR memory. For checkpoint writes and large-tensor reads, this removes a significant data copy and reduces CPU overhead substantially. WEKA’s NeuralMesh supports GPUDirect Storage natively, which is relevant for teams running large-scale pre-training where checkpoint I/O is itself a bottleneck.
WEKA is software-defined and runs on standard x86 or ARM servers. It can run in cloud (AWS, Azure, GCP), on-premises, or in hybrid configurations. A WEKA cluster pools the local NVMe drives of all nodes into a distributed namespace. This matters for cloud deployments: you can deploy WEKA on instances with local NVMe (like AWS i4i or GCP n2-standard with local SSD), and WEKA will stripe across all of that local NVMe while presenting a single POSIX namespace.
The trade-off is cost and complexity. WEKA is proprietary software with per-node or per-capacity licensing. For teams that are already running large GPU clusters with local NVMe, the incremental cost of WEKA licensing may be justified by the reduction in GPU idle time. For smaller clusters or teams that are not yet hitting storage bottlenecks, Lustre (managed or self-hosted) is usually the right starting point.
BeeGFS: Distributed Metadata, HPC Roots
BeeGFS (originally from Fraunhofer Institute, now maintained by ThinkParQ) takes a middle path between Lustre’s traditional MDS/OSS separation and WEKA’s fully distributed approach. BeeGFS distributes metadata by directory: each directory is assigned to one of the available metadata servers, and that server manages the metadata for all files directly under that directory. Data is striped across storage targets, similar to Lustre OSTs.
The advantage of BeeGFS’s per-directory metadata distribution is simplicity and predictable locality. If you structure your dataset with one directory per class or per data source, metadata for each logical grouping stays on one server. The disadvantage is that this is not fully distributed in the way WEKA is: a single hot directory (the root of a dataset you read sequentially) can still saturate its assigned metadata server.
BeeGFS uses a dual licensing model: the client is GPL v2, while the server runs under ThinkParQ’s proprietary Community or Enterprise license (as of February 2026, the Community Edition transitioned from a straightforward EULA to a formal BeeGFS License Agreement). It is source-available rather than fully open source, but the Community Edition remains free for most use cases, which matters for teams with budget constraints. The documentation is solid (docs.beegfs.io), and BeeGFS 8.4.x (current as of mid-2026) has improved HA options for metadata and storage servers.
For AI workloads specifically, BeeGFS works well when:
- Your dataset is structured into many subdirectories (so metadata load is spread across servers)
- You are reading large files sequentially (the striping benefits apply)
- You need POSIX compliance for existing HPC application code
- Budget or licensing constraints make proprietary options impractical
Where BeeGFS runs into trouble is with workloads that have a flat directory structure with millions of small files. All of those files’ metadata lands on the single server that owns the containing directory, and that server becomes the bottleneck. This is a real issue for datasets like Common Crawl shards or large image collections stored as individual files under a single directory.
The deployment story for BeeGFS in cloud is self-managed. There is no AWS-managed or GCP-managed BeeGFS offering. You deploy it yourself on EC2 instances, GCP VMs, or bare-metal nodes, which means you own the operational burden. For teams that have the HPC operations expertise, this is fine. For teams that do not, the operational overhead is real.
GPUDirect Storage: The Emerging Layer
Before deciding on a parallel file system, it is worth understanding GPUDirect Storage (GDS) and where it changes the calculation. GDS is NVIDIA’s DMA technology that allows a storage stack (NVMe, Lustre, parallel file systems in general) to write directly into GPU memory, bypassing the CPU-managed copy that traditionally happens.
In a standard training checkpoint operation without GDS: the training framework calls a write from GPU memory to a file. The GPU-to-CPU memory copy happens first, then the CPU orchestrates a DMA transfer to the NVMe or network storage. For a model with hundreds of billions of parameters, this memory copy is expensive and CPU-bound.
With GDS, the storage driver can initiate a DMA directly from GPU memory to the storage destination, and the CPU’s role is reduced to coordination rather than data movement. WEKA NeuralMesh supports GDS natively. FSx for Lustre supports GDS with the NVIDIA GDS driver installed on the client side. BeeGFS has GDS support available as well.
Whether GDS actually helps you depends on workload pattern. For pre-training runs with frequent checkpoints (every few thousand steps on a large model), the checkpoint write time is a real cost. For fine-tuning runs with infrequent checkpoints and smaller models, GDS makes less difference. If you are considering WEKA specifically for GDS, profile your actual checkpoint overhead first; you may find the bottleneck is elsewhere.
This is connected to the broader theme of storage-aware GPU cluster networking. The network fabric and the storage layer need to be designed together, not independently. A cluster with fast InfiniBand or RoCE fabric but a slow storage tier will have GPUs waiting on data. A cluster with fast storage but slow network fabric will have GPUs waiting on all-reduce. You need both.
Choosing the Right Option for Your Workload
The honest answer is that the right choice depends on cluster size, workload type, operations capability, and budget. Here is how I think through the decision:
For clusters under 16 nodes running fine-tuning or inference: Managed Lustre (FSx for Lustre or Google Cloud Managed Lustre) is the right starting point. The managed offering removes the operational burden, the POSIX compatibility means existing training code works without modification, and the managed options integrate well with cloud object storage for dataset staging. The per-GB cost is higher than building your own, but for a cluster this size, the operational simplicity is worth it.
For 32 to 128-node clusters running pre-training or continuous pre-training: Evaluate WEKA if you are already using instances with local NVMe. The distributed metadata architecture handles the metadata-intensive checkpoint operations better than traditional Lustre, and the GPUDirect Storage support matters at this scale. If you are not using instances with local NVMe, FSx for Lustre or a self-hosted Lustre cluster on dedicated storage nodes is still a solid choice.
For clusters above 128 nodes where storage I/O is confirmed bottleneck: At this scale, the per-node licensing cost of WEKA or DDN’s Lustre distributions may be justified by the reduction in GPU idle time. Self-hosted Lustre with Lustre DNE (for metadata sharding) is viable but requires a dedicated HPC storage team to operate. BeeGFS is viable if your dataset structure distributes metadata load across directories and you have HPC operations expertise.
For ML inference (not training): Parallel file systems are often overkill. Inference workloads typically load a model once at startup, then serve from GPU memory. Object storage or standard block storage is usually sufficient unless you are loading multiple models dynamically in response to traffic patterns. For inference infrastructure specifics, see the guide on LLM inference engines.
For fine-tuning with LoRA or QLoRA on single-node setups: Local NVMe is the right answer. If your entire fine-tuning dataset fits on one node’s local storage, a parallel file system adds complexity and cost for no benefit. See the LLM fine-tuning infrastructure guide for how to structure these workflows.
The AI FinOps perspective matters here too. Every hour your GPUs are idle because storage is the bottleneck is money wasted at GPU prices. Sizing up your storage tier from the budget that was going to sit idle in wasted GPU compute is usually a straightforward optimization.
Dataset Format: Often More Important Than File System Choice
Before you go provision a new parallel file system, audit your dataset format. The choice of data format can make an enormous difference in storage throughput, sometimes more than the choice of file system.
The pathological case is storing training data as millions of individual small files. ImageNet-style directory layouts with one JPEG per sample, or HuggingFace dataset formats that create one file per row in some use cases, can stress even the best parallel file system’s metadata layer. Every training job that reads a random batch needs to open millions of individual file handles, issue millions of stat() calls, and coordinate metadata across the cluster.
The fix is typically to re-pack the dataset into a streaming-friendly format: WebDataset (.tar shards), TFDS (TensorFlow Dataset format with tfrecord shards), or HDF5 for structured scientific datasets. These formats pack many samples into a single file, so the file system sees sequential reads of large files rather than random reads of millions of small ones. This can make a mediocre storage setup perform as well as an expensive parallel file system, or let a parallel file system perform at its theoretical peak rather than its metadata-limited ceiling.
I have seen teams invest significantly in parallel file system infrastructure and then recover comparable performance simply by converting their dataset from a flat JPEG layout to WebDataset shards. Profile before you provision.
Monitoring and Tuning
Once you have a parallel file system deployed, you need visibility into whether storage is actually the bottleneck and where the inefficiency lives.
For Lustre, the primary tool is lfs jobstats, which shows per-job I/O statistics. You can see read/write throughput, metadata operation counts, and identify which jobs are generating excessive metadata load. The lctl get_param llite.*.stats on the client side shows aggregate read/write bytes and operation counts. For managed FSx for Lustre, CloudWatch metrics expose throughput and IOPS at the file system level, though per-job granularity requires client-side collection.
For WEKA, the WEKA Management System (WMS) provides a dashboard with per-client, per-volume, and aggregate I/O metrics. The WEKA CLI (weka stats) can show real-time throughput and operation rates. WEKA also integrates with Prometheus and Grafana for alerting on throughput drops or metadata latency spikes.
For BeeGFS, the beegfs-ctl --serverstats command shows per-server I/O statistics, and beegfs-mon (the BeeGFS monitoring service) can ship metrics to standard time-series databases.
Beyond the file system itself, correlate storage metrics with GPU utilization from DCGM (NVIDIA Data Center GPU Manager) or AMD ROCm SMI. If you see GPU utilization dropping in a regular pattern that aligns with checkpoint writes or data loading phases, storage is the culprit. This kind of joint storage-and-GPU observability is something I push teams to set up from day one, because it makes debugging training job inefficiencies dramatically faster than reconstructing the timeline after the fact from scattered logs.
The Kubernetes persistent storage layer is also relevant if you are running training jobs on Kubernetes. CSI drivers exist for both Lustre and WEKA that expose parallel file systems as PersistentVolumes in a cluster, letting you use standard Kubernetes storage abstractions while still getting parallel file system performance for the underlying data.
The Cloud vs Self-Hosted Decision
For most teams moving into AI infrastructure, managed cloud offerings (FSx for Lustre, Google Cloud Managed Lustre) are the right starting point. The operational simplicity is genuinely valuable, and the performance tiers available from managed offerings are sufficient for most training workloads at medium scale.
The case for self-hosted parallel file systems grows when:
- You are running at a scale where the per-GB or per-TiB cost of managed offerings is material, and you have the operations expertise to run the cluster safely.
- Your workload has specific features (GPUDirect Storage, custom Lustre configuration, WEKA’s composable clusters for multi-tenant isolation) that managed offerings do not expose.
- You are repatriating workloads to bare metal cloud or on-premises and the managed cloud options are not available in your target environment.
The parallel file system space intersects heavily with the broader block vs object vs file storage decision. Object storage remains the right answer for cold datasets, model artifacts in a model registry, and any data that does not need to be read at training-time throughput. Parallel file systems are the hot tier that sits between your object store and your GPU compute.
A pattern that works well at scale: keep the canonical dataset in object storage (cheap, durable, infinitely scalable), use the parallel file system as a hot staging area for the current training run’s data, and automate the import from object storage to the parallel file system at the start of each run. Both FSx for Lustre’s S3 data repository feature and WEKA’s S3-compatible gateway support this kind of tiered workflow.
What the Next Two Years Look Like
The parallel file system space is moving fast because AI infrastructure spending is large enough to fund genuine innovation. A few trends worth watching:
GPUDirect RDMA for storage (a separate concept from GPUDirect Storage) is gaining traction, where the storage network fabric is the same RDMA fabric as the GPU interconnect, eliminating the storage-to-host-to-GPU path entirely. WEKA supports this in configurations where the NVMe devices are on the same nodes as the GPUs. DDN’s EXAScaler and IBM Spectrum Scale have similar capabilities for large-scale deployments.
The composable disaggregated infrastructure model, where storage and compute are pooled separately and connected over ultra-fast network fabrics, is pushing parallel file systems toward a network-attached NVMe model (NVMe-oF). The DPU and SmartNIC layer is relevant here: offloading storage protocol processing to DPUs reduces CPU overhead on the compute nodes, which is significant when you are trying to keep every CPU cycle available for training.
Managed offerings will continue to improve. Google’s 10 TB/s Managed Lustre claim (as of Google Cloud Next ‘26) represents a meaningful step above what was available 18 months ago. The hyperscalers have strong incentive to compete on storage performance for AI workloads, which means the managed option gap versus self-hosted will continue to narrow.
The teams that build a proper understanding of their storage bottleneck before provisioning are the ones who get the most from their GPU spend. Storage is never the glamorous part of an AI infrastructure build, but it is frequently the part that determines whether the glamorous GPU investment actually performs.
For related infrastructure topics, the GPU cloud infrastructure guide covers the compute side of AI cluster design, and fractional GPU sharing with MIG and time-slicing is relevant for teams running multiple smaller jobs on shared infrastructure.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.