Introduction

Managing massive datasets in a data lake often feels like a gamble, where a single erroneous script can wipe out petabytes of critical information without a clear path to recovery. lakeFS solves this by bringing software engineering best practices to data management, allowing teams to treat their data as code. With over 5.5k GitHub stars, lakeFS transforms object storage into a versioned repository, enabling developers to isolate changes, test ETL pipelines in production-like environments, and roll back mistakes instantly. It effectively replaces the need for expensive and slow data duplication by introducing zero-copy branching to the data lake.

What Is lakeFS?

lakeFS is an open-source data version control tool that transforms object storage into Git-like repositories for target users such as data engineers, ML researchers, and DevOps professionals. Built primarily in Go, it provides a logical layer over existing object stores like AWS S3, Azure Blob Storage, and Google Cloud Storage (GCS), without requiring the data to be moved or copied. It is licensed under the Apache 2.0 License.

The project describes itself as “Git for data,” providing the same semantics—branching, committing, and merging—that developers are already familiar with from source code management. By managing pointers and metadata rather than the data itself, lakeFS allows users to create isolated environments for data experimentation and production-grade governance.

Why lakeFS Matters

Before lakeFS, the primary way to test changes to a data lake was to create physical copies of the data—a process that was prohibitively expensive and time-consuming when dealing with petabytes of information. This often led to teams running risky ETL jobs directly in production, which could corrupt data and break downstream dashboards or ML models.

lakeFS fills this gap by introducing zero-copy branching. This means a developer can create a branch of the entire data lake in seconds, regardless of the size of the dataset. This enables a “Dev/Test” workflow for data: developers can run a new ETL pipeline on an isolated branch, validate the results using data quality checks, and then merge the changes into the main branch atomically. If the results are poor, the branch is simply deleted, and production remains untouched.

Furthermore, lakeFS ensures reproducibility. In ML workflows, being able to refer to a specific commit ID or tag allows a data scientist to exactly reproduce a model’s training set, which is critical for auditing, debugging, and regulatory compliance.

Key Features

• Zero-Copy Branching: Create isolated versions of your data lake in seconds without duplicating the actual data. This allows for safe experimentation and testing of ETL pipelines without impacting production.
• Atomic Commits and Merges: Ensure that data updates are exposed to consumers atomically. This prevents “partial writes” where consumers see a mix of old and new data during a long-running write operation.
• Format-Agnostic Versioning: Works with any data format (Parquet, CSV, JSON, images, videos) and any object store (S3, Azure, GCS). It does not require a specific table format like Delta Lake or Iceberg to function.
• Time Travel and Rollbacks: Instantly roll back the entire data lake to a previous commit or tag. This provides a safety net against accidental deletions or corrupted data ingestion.
• S3-Compatible API: lakeFS is API-compatible with S3, meaning it works seamlessly with existing tools like Spark, Presto, AWS Athena, and DuckDB without requiring significant code changes.
• Configurable Garbage Collection: Manage storage costs by automatically removing unreferenced objects that are no longer needed by any branch or commit.
• Role-Based Access Control (RBAC): In the Enterprise version, lakeFS provides granular control over who can create branches, commit changes, or merge data into production.
• Integration with Data Frameworks: Seamlessly integrates with popular libraries such as Pandas, Polars, and HuggingFace Datasets via the lakeFS-spec implementation.

How lakeFS Compares

While DVC is an excellent tool for data scientists managing individual experiments and local model reproducibility, lakeFS is designed for the enterprise data lake. Where DVC tracks metadata for files, lakeFS acts as a proxy layer that manages the entire object store, allowing for instant branching of petabytes of data. This makes lakeFS the superior choice for teams that need to manage production data pipelines and governance at scale.

Compared to Delta Lake, lakeFS operates at a higher level of abstraction. Delta Lake provides ACID transactions and schema enforcement for specific table formats (Parquet), whereas lakeFS provides versioning for the entire bucket, including unstructured data like images or logs. If your team only uses structured tabular data in Spark, Delta Lake is a powerful choice; however, if you need a unified versioning system for all data types across your entire data lake, lakeFS is the correct tool.

Getting Started: Installation

lakeFS can be deployed in several ways depending on your environment and requirements.

Quickstart via Python

The fastest way to try lakeFS locally is using the Python package:

pip install lakefs
python -m lakefs.quickstart

Once running, you can access the web UI at http://127.0.0.1:8000/.

Docker Installation

For a more consistent environment, you can run lakeFS as a container:

docker run --pull always \
  --name lakefs \
  -p 8000:8000 \
  treeverse/lakefs:latest \
  run --quickstart

Kubernetes Deployment

For production-grade installations, lakeFS provides official Helm charts for deployment on Kubernetes clusters.

How to Use lakeFS

The basic workflow in lakeFS mirrors the Git workflow. You start by creating a repository that points to an existing object store bucket. Once the repository is created, you lakeFS manages the metadata and pointers to the objects in that bucket.

To begin, you create a branch from main. This branch is a zero-copy clone of the entire data lake. You then use your existing data tools (Spark, Pandas, etc.) to write new data to this branch. After validating the data quality using your own tests or hooks, youC an atomically merge the branch into main, making the changes visible to all production consumers.

If you are using the lakeFS CLI (lakectl), you common command patterns include:

# Create a branch
lakectl branch create my-feature-branch

# Commit changes to the branch
lakectl commit branch my-feature-branch -m "Updated ETL pipeline"

# Merge branch into main
lakectl merge my-feature-branch main

Code Examples

The following examples demonstrate how to interact with lakeFS using Python, leveraging the lakeFS-spec implementation for seamless integration with data science libraries.

Example 1: Basic File Operations

This example shows how to upload a file and create a commit in lakeFS using the lakefs-spec library.

from lakefs_spec import LakeFSFileSystem

REPO, BRANCH = "my-repo", "main"
fs = LakeFSFileSystem()

# Upload a file to the lakeFS repository
fs.put("lakefs://my-repo/main/data.csv", data=b"Hello lakeFS!")

# Create a commit to version the data
fs.commit(branch=BRANCH, message="Initial data upload")

This code snippet allows you to version your data files directly from a Python script, treating the data as a first-class citizen in your version control system.

Example 2: Branching and Data Validation

This example demonstrates the typical “Dev/Test” workflow: creating a branch, writing data, and merging.

from lakefs_spec import LakeFSFileSystem

fs = LakeFSFileSystem()

# 1. Create a development branch from main
fs.branch(name="dev-branch", source= "main")

# 2. Write new data to the dev branch
fs.put("lakefs://my-repo/dev-branch/processed_data.parquet", data=b"some binary data")

# 3. Perform data quality checks (e.g., using Great Expectations)
# If checks pass, merge the dev branch into main
if data_quality_passed:
    fs.merge(source="dev-branch", destination="main")
else:
    fs.delete_branch("dev-branch")

This example highlights the lakeFS value proposition: the ability to isolate changes and prevent corrupted data from ever reaching production.

Real-World Use Cases

lakeFS is particularly effective in scenarios where data integrity and reproducibility are required at scale.

• ETL Testing in Production: A data engineer can create a branch of the production data lake, run a new version of an ETL pipeline, and validate the results against the production data without impacting any downstream consumers. This eliminates the the risk of “breaking the lake.”
• ML Model Reproducibility: An ML researcher can tag a specific commit of the data lake at the time of model training. By referencing this commit ID, the researcher can exactly reproduce the training set, ensuring that the same model is trained on the same data version, which is critical for auditing and regulatory compliance.
• Data Recovery and Rollbacks: A DevOps engineer can instantly roll back the entire data lake to a previous known-good state if a corrupted data ingestion process accidentally deletes or overwrites critical files. This provides a disaster recovery mechanism that is virtually instantaneous regardless of the data size.
• Curation of Large Datasets: For teams curating massive image or video datasets for AI training, lakeFS allows multiple researchers to work on isolated branches, each refining the same dataset, and then merging their curated curated data into a main branch.

Contributing to lakeFS

lakeFS is an open-source project that encourages community contributions. To get started, you can explore the rest of the codebase and check for “good first issues” on GitHub.

The project maintains a clear code of conduct and requires contributors to sign a Contributor License Agreement (CLA) before their first pull request. Contributions should be associated with an existing GitHub issue to ensure the project maintainers maintain a clear roadmap.

For those interested in contributing to the documentation, lakeFS follows the Diátaxis approach, focusing on Tutorials, How-To guides, and Reference material. Documentation contributions are written in Markdown and can be submitted via pull requests.

Community and Support

lakeFS provides several official channels for support and collaboration. The primary hub for community interaction is the official Slack community, where data engineers and practitioners can share best practices and discover new features.

In addition to Slack, the project maintains an active GitHub Discussions forum for more structured conversations and a GitHub repository for reporting bugs and requesting features. The official documentation site is a comprehensive resource for the lakeFS ecosystem.

Conclusion

lakeFS is the right choice for organizations that have outgrown simple data versioning and need a scalable, governance-focused approach to managing their data lake. It is particularly powerful for teams that is using a multi-cloud or multi-object store strategy, as it is format-agnostic and works across AWS, Azure, and GCP.

While the project has a steep learning curve for those unfamiliar with Git semantics, it is a recommended tool for any team that is struggling with data corruption, lack of reproducibility in ML, or the expensive cost of data duplication for testing. Star the repo, try the quickstart, and join the community to start treating your data as code.

Resources

• lakeFS GitHub Repository
• Official lakeFS Documentation
• lakeFS Slack Community