Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Prefect | 12,110 | 1 | 70 | 3 hours ago | 162 | July 05, 2022 | 520 | apache-2.0 | Python | |
The easiest way to orchestrate and observe your data pipelines | ||||||||||
Great_expectations | 8,441 | 3 | 33 | 3 hours ago | 179 | June 30, 2022 | 164 | apache-2.0 | Python | |
Always know what to expect from your data. | ||||||||||
Dagster | 7,609 | 2 | 89 | 4 hours ago | 495 | July 06, 2022 | 1,690 | apache-2.0 | Python | |
An orchestration platform for the development, production, and observation of data assets. | ||||||||||
Pachyderm | 5,931 | 1 | 10 hours ago | 220 | September 22, 2022 | 899 | apache-2.0 | Go | ||
Data-Centric Pipelines and Data Versioning | ||||||||||
Mage Ai | 4,815 | 9 hours ago | 9 | June 27, 2022 | 83 | apache-2.0 | Python | |||
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data. | ||||||||||
Orchest | 3,876 | 3 days ago | 14 | April 06, 2022 | 125 | apache-2.0 | TypeScript | |||
Build data pipelines, the easy way 🛠️ | ||||||||||
Datascienceresources | 3,710 | a month ago | 19 | |||||||
Open Source Data Science Resources. | ||||||||||
Polyaxon | 3,326 | 4 | 11 | 6 hours ago | 334 | June 05, 2022 | 122 | apache-2.0 | ||
MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle | ||||||||||
Pipelines | 3,209 | 2 | 58 | 15 hours ago | 98 | June 02, 2022 | 964 | apache-2.0 | Python | |
Machine Learning Pipelines for Kubeflow | ||||||||||
Ploomber | 3,083 | 5 | 8 days ago | 96 | July 04, 2022 | 121 | apache-2.0 | Python | ||
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️ |
Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.
It is designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.
With Dagster, you declareas Python functionsthe data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date.
Here is an example of a graph of three assets defined in Python:
from dagster import asset
from pandas import DataFrame, read_html, get_dummies
from sklearn.linear_model import LinearRegression
@asset
def country_populations() -> DataFrame:
df = read_html("https://tinyurl.com/mry64ebh")[0]
df.columns = ["country", "continent", "rg", "pop2018", "pop2019", "change"]
df["change"] = df["change"].str.rstrip("%").str.replace("", "-").astype("float")
return df
@asset
def continent_change_model(country_populations: DataFrame) -> LinearRegression:
data = country_populations.dropna(subset=["change"])
return LinearRegression().fit(get_dummies(data[["continent"]]), data["change"])
@asset
def continent_stats(country_populations: DataFrame, continent_change_model: LinearRegression) -> DataFrame:
result = country_populations.groupby("continent").sum()
result["pop_change_factor"] = continent_change_model.coef_
return result
The graph loaded into Dagster's web UI:
Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.
If you're new to Dagster, we recommend reading about its core concepts or learning with the hands-on tutorial.
Dagster is available on PyPI and officially supports Python 3.7+.
pip install dagster dagit
This installs two modules:
Running on Using a Mac with an M1 or M2 chip? Check the install details here.
You can find the full Dagster documentation here, including the 'getting started' guide.
Identify the key assets you need to create using a declarative approach, or you can focus on running basic tasks. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.
Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.
Maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.
Dagster provides a growing library of integrations for todays most popular data tools. Integrate with the tools you already use, and deploy to your infrastructure.
Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.
Join our community here:
For details on contributing or running the project for development, check out our contributing guide.
Dagster is Apache 2.0 licensed.