Dagster

An orchestration platform for the development, production, and observation of data assets.
Alternatives To Dagster
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Prefect12,850113811 hours ago225August 01, 2023550apache-2.0Python
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
Tpot9,213402015 days ago61January 06, 2021281lgpl-3.0Python
A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.
Great_expectations8,7853511 hours ago236August 04, 2023162apache-2.0Python
Always know what to expect from your data.
Dagster8,4974111 hours ago105September 30, 20221,993apache-2.0Python
An orchestration platform for the development, production, and observation of data assets.
Pachyderm5,976111 hours ago504August 04, 2023881apache-2.0Go
Data-Centric Pipelines and Data Versioning
Mage Ai5,527
12 hours ago278August 08, 2023142apache-2.0Python
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
Orchest3,876
4 months ago19December 13, 2022125apache-2.0TypeScript
Build data pipelines, the easy way 🛠️
Datascienceresources3,826
a month ago20
Open Source Data Science Resources.
Polyaxon3,38541215 hours ago377August 14, 2023122apache-2.0
MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
Pipelines3,29027116 hours ago125July 28, 20231,039apache-2.0Python
Machine Learning Pipelines for Kubeflow
Alternatives To Dagster
Select To Compare


Alternative Project Comparisons
Readme
dagster logo

Remember to star the Dagster GitHub repo for future reference.

Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

It is designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.

With Dagster, you declareas Python functionsthe data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date.

Here is an example of a graph of three assets defined in Python:

from dagster import asset
from pandas import DataFrame, read_html, get_dummies
from sklearn.linear_model import LinearRegression

@asset
def country_populations() -> DataFrame:
    df = read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "continent", "rg", "pop2018", "pop2019", "change"]
    df["change"] = df["change"].str.rstrip("%").str.replace("", "-").astype("float")
    return df

@asset
def continent_change_model(country_populations: DataFrame) -> LinearRegression:
    data = country_populations.dropna(subset=["change"])
    return LinearRegression().fit(get_dummies(data[["continent"]]), data["change"])

@asset
def continent_stats(country_populations: DataFrame, continent_change_model: LinearRegression) -> DataFrame:
    result = country_populations.groupby("continent").sum()
    result["pop_change_factor"] = continent_change_model.coef_
    return result

The graph loaded into Dagster's web UI:

An example asset graph as rendered in the Dagster UI

Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.

Quick Start:

If you're new to Dagster, we recommend reading about its core concepts or learning with the hands-on tutorial.

Dagster is available on PyPI and officially supports Python 3.8+.

pip install dagster dagster-webserver

This installs two packages:

  • dagster: The core programming model.
  • dagster-webserver: The server that hosts Dagster's web UI for developing and operating Dagster jobs and assets.

Running on Using a Mac with an M1 or M2 chip? Check the install details here.

Documentation

You can find the full Dagster documentation here, including the 'getting started' guide.


Key Features:

image

Dagster as a productivity platform

Identify the key assets you need to create using a declarative approach, or you can focus on running basic tasks. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.

Dagster as a robust orchestration engine

Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.

Dagster as a unified control plane

Maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.


Master the Modern Data Stack with integrations

Dagster provides a growing library of integrations for todays most popular data tools. Integrate with the tools you already use, and deploy to your infrastructure.


image

Community

Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.

Join our community here:

Contributing

For details on contributing or running the project for development, check out our contributing guide.

License

Dagster is Apache 2.0 licensed.

Popular Data Science Projects
Popular Pipeline Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Pipeline
Data Science
Spark
Etl
Snowflake
Data Engineering
Datadog