lakeFS - Data version control for your data lake | Git for data
Alternatives To Lakefs
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Data Engineering Zoomcamp13,734
7 days ago48Jupyter Notebook
Free Data Engineering course!
Prefect12,0871707 hours ago162July 05, 2022514apache-2.0Python
The easiest way to orchestrate and observe your data pipelines
Lakefs3,42418 hours ago62June 15, 2022561apache-2.0Go
lakeFS - Data version control for your data lake | Git for data
Everything Tech372
a year agoapache-2.0Go
A collection of online resources to help you on your Tech journey.
2 months ago33otherJavaScript
Dataplane is an Airflow inspired data platform with additional data mesh capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
Movalytics Data Warehouse74
3 years agoPython
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
3 months ago53April 01, 202239apache-2.0Python
Containerized distributed programming framework for Python
4 months ago7Python
This repo contains commands that data engineers use in day to day work.
2 years agoJava
A Spring Boot Camel boilerplate that aims to consume events from Apache Kafka, process it and send to a PostgreSQL database.
8 months agomitGo
rtdl makes it easy to build and maintain a real-time data lake
Alternatives To Lakefs
Select To Compare

Alternative Project Comparisons

Apache License Go tests status Node tests status Integration tests status Docs Preview & Link Check status Artifact HUB code of conduct

lakeFS is Data Version Control (Git for Data)

lakeFS is an open-source tool that transforms your object storage into a Git-like repository. It enables you to manage your data lake the way you manage your code.

With lakeFS you can build repeatable, atomic, and versioned data lake operations - from complex ETL jobs to data science and analytics.

lakeFS supports AWS S3, Azure Blob Storage, and Google Cloud Storage as its underlying storage service. It is API compatible with S3 and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, DuckDB, and Presto.

For more information, see the documentation.

Getting Started

You can spin up a standalone sandbox instance of lakeFS using Docker:

docker run --pull always \
		   --name lakefs \
		   -p 8000:8000 \
		   treeverse/lakefs:latest \
		   run --local-settings

Once you've got lakeFS running, open in your web browser.


👉🏻 For a hands-on walk through of the core functionality in lakeFS head over to the quickstart to jump right in!

Make sure to also have a look at the lakeFS samples. These are a rich resource of examples of end-to-end applications that you can build with lakeFS.

Why Do I Need lakeFS?

ETL Testing with Isolated Dev/Test Environment

When working with a data lake, it’s useful to have replicas of your production environment. These replicas allow you to test these ETLs and understand changes to your data without impacting downstream data consumers.

Running ETL and transformation jobs directly in production without proper ETL Testing is a guaranteed way to have data issues flow into dashboards, ML models, and other consumers sooner or later. The most common approach to avoid making changes directly in production is to create and maintain multiple data environments and perform ETL testing on them. Dev environment to develop the data pipelines and test environment where pipeline changes are tested before pushing it to production. With lakeFS you can create branches, and get a copy of the full production data, without copying anything. This enables a faster and easier process of ETL testing.


Data changes frequently. This makes the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state.

This has a negative impact on the work, as it becomes hard to:

  • Debug a data issue.
  • Validate machine learning training accuracy (re-running a model over different data gives different results). Comply with data audits.

In comparison, lakeFS exposes a Git-like interface to data that allows keeping track of more than just the current state of data. This makes reproducing its state at any point in time straightforward.

CI/CD for Data

Data pipelines feed processed data from data lakes to downstream consumers like business dashboards and machine learning models. As more and more organizations rely on data to enable business critical decisions, data reliability and trust are of paramount concern. Thus, it’s important to ensure that production data adheres to the data governance policies of businesses. These data governance requirements can be as simple as a file format validation, schema check, or an exhaustive PII(Personally Identifiable Information) data removal from all of organization’s data.

Thus, to ensure the quality and reliability at each stage of the data lifecycle, data quality gates need to be implemented. That is, we need to run Continuous Integration(CI) tests on the data, and only if data governance requirements are met can the data can be promoted to production for business use.

Everytime there is an update to production data, the best practice would be to run CI tests and then promote(deploy) the data to production. With lakeFS you can create hooks that make sure that only data that passed these tests will become part of production.


A rollback operation is used to to fix critical data errors immediately.

What is a critical data error? Think of a situation where erroneous or misformatted data causes a signficant issue with an important service or function. In such situations, the first thing to do is stop the bleeding.

Rolling back returns data to a state in the past, before the error was present. You might not be showing all the latest data after a rollback, but at least you aren’t showing incorrect data or raising errors. Since lakeFS provides versions of the data without making copies of the data, you can time travel between versions and roll back to the version of the data before the error was presented.


Stay up to date and get lakeFS support via:

More information


lakeFS is completely free and open-source and licensed under the Apache 2.0 License.

Who Uses lakeFS?

lakeFS is used by numerous companies, including those below. If you use lakeFS and would like to be included here please open a PR.

  • AirAsia
  • APEX Global
  • AppsFlyer
  • Auburn University
  • BAE Systems
  • Bureau of Labor Statistics
  • Cambridge Consultants
  • Connor, Clark & Lunn Financial Group
  • Context Labs Bv
  • Daimler Truck
  • Enigma
  • Ford Motor Company
  • Generali
  • Giesecke+Devrient
  • greehill
  • Karius
  • Lockheed Martin
  • Luxonis
  • Mixpeek
  • Netflix
  • Paige
  • Pollinate
  • Proton Technologies AG
  • ProtonMail
  • Renaissance Computing Institute
  • RHEA Group
  • RMS
  • Sensum
  • Similarweb
  • State Street Global Advisors
  • Terramera
  • Tredence
  • Volvo Cars
  • Webiks
  • Windward
  • Woven by Toyota
Popular Data Engineering Projects
Popular Docker Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Aws S3
Apache Spark
Data Engineering
Azure Storage
Object Storage
Google Cloud Storage