Pipelines
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Jina | 17,896 |  | 2 | a day ago | 2,019 | July 06, 2022 | 39 | apache-2.0 | Python | |
| 🔮 Build multimodal AI services via cloud native technologies · Neural Search · Generative AI · Cloud Native | ||||||||||
| Kubeflow | 12,397 | 2 | a day ago | 112 | April 13, 2021 | 397 | apache-2.0 | TypeScript | ||
| Machine Learning Toolkit for Kubernetes | ||||||||||
| Tpot | 8,989 |  | 40 | 18 | 4 days ago | 60 | January 06, 2021 | 284 | lgpl-3.0 | Python |
| A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming. | ||||||||||
| Kedro | 8,206 |  | 8 | 32 | 17 hours ago | 35 | May 09, 2022 | 283 | apache-2.0 | Python |
| A Python framework for creating reproducible, maintainable and modular data science code. | ||||||||||
| Stanza | 6,546 |  | 2 | 68 | 20 hours ago | 17 | April 23, 2022 | 73 | other | Python |
| Official Stanford NLP Python Library for Many Human Languages | ||||||||||
| Augmentor | 4,849 |  | 21 | 8 | 2 months ago | 22 | April 27, 2022 | 132 | mit | Python |
| Image augmentation library in Python for machine learning. | ||||||||||
| Clearml | 4,234 |  | 8 | a day ago | 93 | July 04, 2022 | 309 | apache-2.0 | Python | |
| ClearML - Auto-Magical CI/CD to streamline your ML workflow. Experiment Manager, MLOps and Data-Management | ||||||||||
| Deeplearningproject | 4,043 | 3 years ago | 3 | mit | HTML | |||||
| An in-depth machine learning tutorial introducing readers to a whole machine learning pipeline from scratch. | ||||||||||
| Orchest | 3,773 | 7 days ago | 14 | April 06, 2022 | 124 | agpl-3.0 | Python | |||
| Build data pipelines, the easy way 🛠️ | ||||||||||
| Mage Ai | 3,611 | 18 hours ago | 9 | June 27, 2022 | 52 | apache-2.0 | Python | |||
| 🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data. | ||||||||||
Overview of the Kubeflow pipelines service
Kubeflow is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on Kubernetes simple, portable, and scalable.
Kubeflow pipelines are reusable end-to-end ML workflows built using the Kubeflow Pipelines SDK.
The Kubeflow pipelines service has the following goals:
- End to end orchestration: enabling and simplifying the orchestration of end to end machine learning pipelines
- Easy experimentation: making it easy for you to try numerous ideas and techniques, and manage your various trials/experiments.
- Easy re-use: enabling you to re-use components and pipelines to quickly cobble together end to end solutions, without having to re-build each time.
Installation
-
Install Kubeflow Pipelines from choices described in Installation Options for Kubeflow Pipelines.
-
The Docker container runtime has been deprecated on Kubernetes 1.20+. Kubeflow Pipelines has switched to use Emissary Executor by default from Kubeflow Pipelines 1.8. Emissary executor is Container runtime agnostic, meaning you are able to run Kubeflow Pipelines on Kubernetes cluster with any Container runtimes.
Documentation
Get started with your first pipeline and read further information in the Kubeflow Pipelines overview.
See the various ways you can use the Kubeflow Pipelines SDK.
See the Kubeflow Pipelines API doc for API specification.
Consult the Python SDK reference docs when writing pipelines using the Python SDK.
Refer to the versioning policy and feature stages documentation for more information about how we manage versions and feature stages (such as Alpha, Beta, and Stable).
Contributing to Kubeflow Pipelines
Before you start contributing to Kubeflow Pipelines, read the guidelines in How to Contribute. To learn how to build and deploy Kubeflow Pipelines from source code, read the developer guide.
Kubeflow Pipelines Community Meeting
The meeting is happening every other Wed 10-11AM (PST) Calendar Invite or Join Meeting Directly
Kubeflow Pipelines Slack Channel
Blog posts
- Getting started with Kubeflow Pipelines (By Amy Unruh)
- How to create and deploy a Kubeflow Machine Learning Pipeline (By Lak Lakshmanan)
- Kubeflow Pipelines meets Tekton (By Animesh Singh)
Acknowledgments
Kubeflow pipelines uses Argo Workflows by default under the hood to orchestrate Kubernetes resources. The Argo community has been very supportive and we are very grateful. Additionally there is Tekton backend available as well. To access it, please refer to Kubeflow Pipelines with Tekton repository.
