|Project Name||Stars||Downloads||Repos Using This||Packages Using This||Most Recent Commit||Total Releases||Latest Release||Open Issues||License||Language|
|Data Engineering Zoomcamp||13,734||7 days ago||48||Jupyter Notebook|
|Free Data Engineering course!|
|Prefect||12,087||1||70||7 hours ago||162||July 05, 2022||514||apache-2.0||Python|
|The easiest way to orchestrate and observe your data pipelines|
|Lakefs||3,424||1||8 hours ago||62||June 15, 2022||561||apache-2.0||Go|
|lakeFS - Data version control for your data lake | Git for data|
|Everything Tech||372||a year ago||apache-2.0||Go|
|A collection of online resources to help you on your Tech journey.|
|Dataplane is an Airflow inspired data platform with additional data mesh capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.|
|Movalytics Data Warehouse||74||3 years ago||Python|
|Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow|
|Cowait||54||3 months ago||53||April 01, 2022||39||apache-2.0||Python|
|Containerized distributed programming framework for Python|
|Towardsdataengineering||52||4 months ago||7||Python|
|This repo contains commands that data engineers use in day to day work.|
|Camelboilerplate||42||2 years ago||Java|
|A Spring Boot Camel boilerplate that aims to consume events from Apache Kafka, process it and send to a PostgreSQL database.|
|Rtdl||39||8 months ago||mit||Go|
|rtdl makes it easy to build and maintain a real-time data lake|
lakeFS is an open-source tool that transforms your object storage into a Git-like repository. It enables you to manage your data lake the way you manage your code.
With lakeFS you can build repeatable, atomic, and versioned data lake operations - from complex ETL jobs to data science and analytics.
lakeFS supports AWS S3, Azure Blob Storage, and Google Cloud Storage as its underlying storage service. It is API compatible with S3 and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, DuckDB, and Presto.
For more information, see the documentation.
You can spin up a standalone sandbox instance of lakeFS using Docker:
docker run --pull always \ --name lakefs \ -p 8000:8000 \ treeverse/lakefs:latest \ run --local-settings
Once you've got lakeFS running, open http://127.0.0.1:8000/ in your web browser.
👉🏻 For a hands-on walk through of the core functionality in lakeFS head over to the quickstart to jump right in!
Make sure to also have a look at the lakeFS samples. These are a rich resource of examples of end-to-end applications that you can build with lakeFS.
When working with a data lake, it’s useful to have replicas of your production environment. These replicas allow you to test these ETLs and understand changes to your data without impacting downstream data consumers.
Running ETL and transformation jobs directly in production without proper ETL Testing is a guaranteed way to have data issues flow into dashboards, ML models, and other consumers sooner or later. The most common approach to avoid making changes directly in production is to create and maintain multiple data environments and perform ETL testing on them. Dev environment to develop the data pipelines and test environment where pipeline changes are tested before pushing it to production. With lakeFS you can create branches, and get a copy of the full production data, without copying anything. This enables a faster and easier process of ETL testing.
Data changes frequently. This makes the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state.
This has a negative impact on the work, as it becomes hard to:
In comparison, lakeFS exposes a Git-like interface to data that allows keeping track of more than just the current state of data. This makes reproducing its state at any point in time straightforward.
Data pipelines feed processed data from data lakes to downstream consumers like business dashboards and machine learning models. As more and more organizations rely on data to enable business critical decisions, data reliability and trust are of paramount concern. Thus, it’s important to ensure that production data adheres to the data governance policies of businesses. These data governance requirements can be as simple as a file format validation, schema check, or an exhaustive PII(Personally Identifiable Information) data removal from all of organization’s data.
Thus, to ensure the quality and reliability at each stage of the data lifecycle, data quality gates need to be implemented. That is, we need to run Continuous Integration(CI) tests on the data, and only if data governance requirements are met can the data can be promoted to production for business use.
Everytime there is an update to production data, the best practice would be to run CI tests and then promote(deploy) the data to production. With lakeFS you can create hooks that make sure that only data that passed these tests will become part of production.
A rollback operation is used to to fix critical data errors immediately.
What is a critical data error? Think of a situation where erroneous or misformatted data causes a signficant issue with an important service or function. In such situations, the first thing to do is stop the bleeding.
Rolling back returns data to a state in the past, before the error was present. You might not be showing all the latest data after a rollback, but at least you aren’t showing incorrect data or raising errors. Since lakeFS provides versions of the data without making copies of the data, you can time travel between versions and roll back to the version of the data before the error was presented.
Stay up to date and get lakeFS support via:
lakeFS is completely free and open-source and licensed under the Apache 2.0 License.
lakeFS is used by numerous companies, including those below. If you use lakeFS and would like to be included here please open a PR.