Introduction

Training large-scale deep learning models often hits a critical bottleneck: the data loading pipeline. When datasets grow to terabytes or petabytes, downloading the entire dataset to a local disk before training begins is impractical and costly. MosaicML Streaming is an open-source data streaming library designed to solve this by allowing PyTorch users to stream training data directly from cloud-based object stores. With over 1.5k GitHub stars, it transforms the traditional “download-then-train” workflow into a seamless, cloud-native process that maximizes GPU utilization and minimizes storage overhead.

What Is MosaicML Streaming?

MosaicML Streaming is a PyTorch-compatible dataset library that enables the efficient streaming of training data from cloud-based object stores (such as AWS S3, Google Cloud Storage, and Azure Blob Storage) to training clusters. It is designed as a drop-in replacement for the IterableDataset class in PyTorch, making it easy for developers to integrate into existing workflows without rewriting their entire data pipeline.

Maintained by the team at MosaicML (now part of Databricks), the project is released under the Apache License 2.0. It focuses on providing high-performance, accurate streaming that ensures data is delivered to the GPU as fast as the model can consume it, regardless of where the data is physically stored.

Why MosaicML Streaming Matters

In the era of foundation models, datasets are too large to fit on local NVMe drives. The traditional approach of copying data from S3 to a local disk creates a massive “time-to-train” delay and requires expensive persistent storage on every training node. MosaicML Streaming eliminates this requirement by streaming shards of data on-demand.

Beyond simple streaming, the library addresses the “correctness” problem in distributed training. In many streaming implementations, changing the number of GPUs or nodes can lead to samples being skipped or duplicated. MosaicML Streaming implements Elastic Determinism, ensuring that the sample order remains identical regardless of the cluster size, which is critical for reproducibility and debugging loss spikes.

Furthermore, it enables Instant Mid-Epoch Resumption. If a training run crashes, you can resume from a checkpoint in seconds rather than hours, as the library tracks exactly which samples have been seen, avoiding the need to re-download or replay the entire dataset from the start of the epoch.

Key Features

• Elastic Determinism: Ensures samples are delivered in the same order regardless of the number of GPUs, nodes, or CPU workers used, enabling perfect reproducibility across different cluster configurations.
• Instant Mid-Epoch Resumption: Allows training to resume almost immediately after a failure by tracking the exact sample index, eliminating the need to replay the dataset.
• Cloud-Native Storage Support: Native integration with AWS S3, OCI, GCS, Azure, Databricks UC Volume, and any S3-compatible object store (e.g., Cloudflare R2, Backblaze B2).
• Mosaic Data Shard (MDS) Format: A specialized binary format that encodes any Python object, optimized for high-throughput streaming and efficient random access.
• Disk Usage Limits: Dynamically deletes the least recently used (LRU) shards to keep the local cache under a specified limit (e.g., cache_limit='100gb'), preventing disk-full errors on ephemeral clusters.
• Random Access: Supports accessing specific samples (e.g., dataset[i]) even if the sample hasn’t been downloaded yet, triggering an immediate download of the required shard.
• PyTorch Compatibility: Acts as a drop-in replacement for torch.utils.data.IterableDataset, ensuring seamless integration with standard PyTorch DataLoader.
• Multi-modal Data Support: Compatible with any data type, including images, text, video, and complex multimodal datasets.

How MosaicML Streaming Compares

When comparing MosaicML Streaming to alternatives like WebDataset, the primary differentiator is correctness and reproducibility. While WebDataset is excellent for sequential streaming, it struggles with mid-epoch resumption and deterministic shuffling across varying cluster sizes. MosaicML Streaming solves this by decoupling the sample index from the physical shard location, allowing it to calculate exactly which sample should be seen by which GPU at any given time.

For developers who have previously relied on native PyTorch IterableDataset, the transition is straightforward. However, the native implementation requires the developer to handle the complexities of sharding, cloud authentication, and local caching manually. MosaicML Streaming abstracts these layers, providing a production-ready infrastructure that is optimized for the specific access patterns of deep learning training.

Getting Started: Installation

MosaicML Streaming is available as a Python package and can be installed via pip. It requires Python 3.7 or higher.

Standard Installation

pip install mosaicml-streaming

Installation with Simulator

If you wish to use the Streaming Simulator to analyze your throughput and network usage before running a full training job, install the simulator dependencies:

pip install --upgrade "mosaicml-streaming[simulator]"

Verification

You can verify the installation by running the following command in your terminal:

python -c "import streaming; print(streaming.__version__)"

How to Use MosaicML Streaming

Using the library involves a two-step process: converting your raw data into the Mosaic Data Shard (MDS) format and then loading that data using the StreamingDataset class.

First, you use an MDSWriter to save your samples into shards. This process can be done locally or directly to a cloud bucket. Each sample is serialized and grouped into shards to optimize the download size and the subsequent read speed.

Once the data is sharded, you initialize a StreamingDataset by pointing it to the remote directory (e.g., an S3 bucket) and specifying a local cache directory. The library then handles the background downloading of shards, prefetching of samples, and local cache management automatically.

Code Examples

1. Converting Data to MDS Format

This example shows how to use MDSWriter to convert a simple list of samples into the streaming-optimized MDS format.

from streaming import MDSWriter
import numpy as np
from PIL import Image

# Define the output directory and the data types of the columns
out_root = 'my_dataset_shards'
columns = {'uuid': 'str', 'img': 'jpeg', 'clf': 'int'}

# Create a set of dummy samples
samples = [
    {'uuid': str(i), 'img': Image.fromarray(np.random.randint(0, 256, (32, 32, 3), np.uint8)), 'clf': np.random.randint(10)}
    for i in range(1000)
]

# Write the samples to MDS shards
with MDSWriter(out=out_root, columns=columns, compression='zstd') as out:
    for sample in samples:
        out.write(sample)

2. Loading and Streaming Data

This example demonstrates how to integrate StreamingDataset into a standard PyTorch DataLoader.

from torch.utils.data import DataLoader
from streaming import StreamingDataset

# Remote directory where the MDS shards are stored (S3, GCS, etc.)
remote_dir = 's3://my-bucket/my_dataset_shards'
# Local directory for caching shards during training
local_dir = '/tmp/streaming_cache'

# Initialize the StreamingDataset
dataset = StreamingDataset(local=local_dir, remote=remote_dir, shuffle=True, batch_size=1)

# Create a PyTorch DataLoader
dataloader = DataLoader(dataset, batch_size=32)

for batch in dataloader:
    # Your training loop here
    print(batch)

Real-World Use Cases

MosaicML Streaming is particularly effective in the following scenarios:

• Large-Scale LLM Pre-training: When training on trillions of tokens of text data stored in S3, the library ensures that GPUs are never starved for data, maximizing the efficiency of expensive H100 clusters.
• Multi-modal Foundation Models: For datasets containing millions of high-resolution images or videos, the MDS format and LRU caching prevent the training nodes from running out of disk space while maintaining high throughput.
• Ephemeral Training Clusters: In environments like Kubernetes or AWS SageMaker where nodes are ephemeral and lack persistent storage, the library’s ability to stream data on-demand eliminates the need for expensive EBS volumes.
• Distributed Training on Heterogeneous Hardware: Because of Elastic Determinism, researchers can debug a training run on a small 8-GPU node and perfectly reproduce the exact same sample sequence on a 128-GPU cluster.

Contributing to MosaicML Streaming

The project welcomes contributions from the community. Developers can contribute by reporting bugs via GitHub Issues or submitting Pull Requests for new features or performance optimizations. If you are looking for a place to start, check the issues labeled good first issue to find accessible entry points for new contributors.

The project follows the standard GitHub flow for contributions and adheres to a Code of Conduct to ensure a professional and collaborative environment.

Community and Support

MosaicML Streaming is widely used in the production training of models like BioMedLM and Mosaic Diffusion. It is supported by the team at Databricks Mosaic AI. la community is primarily centered around GitHub Discussions and the official documentation site. For real-time support and interaction, users can join the Community Slack channel provided in the project’s README.

Conclusion

MosaicML Streaming is the right choice for any team training large-scale models where the dataset size exceeds the local storage capacity of the training nodes. By solving the critical problems of cloud-native data loading, elastic determinism, and instant resumption, it removes the data bottleneck from the GPU training loop.

While the library requires a one-time data conversion step to the MDS format, this investment pays off in the highest possible GPU utilization and significantly reduced storage costs. We recommend that researchers and ML engineers start by converting a subset of their data to MDS and testing the throughput with the Streaming Simulator.

Star the repo, try the quickstart, and join the community to optimize your training pipelines.

Resources

• MosaicML Streaming GitHub Repository
• Databricks Mosaic AI Documentation
• Streaming Quickstart Guide