The Top 23 Python Pipeline Open Source Projects

Open source projects categorized as Python Pipeline

Categories > Data Processing > Pipeline

Categories > Programming Languages > Python

Hic Pro ⭐ 346

HiC-Pro: An optimized and flexible pipeline for Hi-C data processing

most recent commit 4 months ago

Melusine ⭐ 335

Melusine is a high-level library for emails classification and feature extraction "dédiée aux courriels français".

total releases 53most recent commit 3 months ago

Pipelines ⭐ 334

Pipelines is a language for high-level data analysis.

most recent commit 4 years ago

Katana Skipper ⭐ 332

Simple and flexible ML workflow engine

dependent packages 2total releases 6most recent commit 6 months ago

Fileconveyor ⭐ 329

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.

most recent commit 7 years ago

Marcel ⭐ 326

A modern shell

total releases 131most recent commit 3 months ago

Pipe2py ⭐ 322

A project to compile Yahoo! Pipes into Python (see it hosted on Google App Engine: http://pipes-engine.appspot.com)

total releases 3most recent commit 4 years ago

Pyscenic ⭐ 321

pySCENIC is a lightning-fast python implementation of the SCENIC pipeline (Single-Cell rEgulatory Network Inference and Clustering) which enables biologists to infer transcription factors, gene regulatory networks and cell types from single-cell RNA-seq data.

dependent packages 6total releases 63most recent commit 9 months ago

Gtex Pipeline ⭐ 317

GTEx & TOPMed data production and analysis pipelines

most recent commit 3 months ago

Neural Pipeline ⭐ 314

Neural networks training pipeline based on PyTorch

total releases 19most recent commit 4 years ago

Lsfm ⭐ 309

Large Scale Facial Model (LSFM) - an automatic pipeline for constructing 3D Morphable Models from large collections of facial meshes

most recent commit 7 years ago

Hypergbm ⭐ 306

A full pipeline AutoML tool for tabular data

dependent packages 1total releases 17most recent commit 4 months ago

Amazon Textract Serverless Large Scale Document Processing ⭐ 302

Process documents at scale using Amazon Textract

most recent commit a year ago

Data Engineering With Python ⭐ 302

Data Engineering with Python, published by Packt

most recent commit a year ago

Sparkflow ⭐ 301

Easy to use library to bring Tensorflow on Apache Spark

total releases 13most recent commit 6 months ago

Cyborg ⭐ 300

Python web scraping framework

most recent commit 6 years ago

Sparktorch ⭐ 297

Train and run Pytorch models on Apache Spark.

total releases 11most recent commit a year ago

Fluids ⭐ 296

Fluid dynamics component of Chemical Engineering Design Library (ChEDL)

dependent packages 10total releases 94most recent commit 6 months ago

Gokart ⭐ 296

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.

dependent packages 4total releases 72most recent commit 3 months ago

Fondant ⭐ 293

Production-ready data processing made easy and shareable

total releases 24most recent commit 3 months ago

Atac Seq Pipeline ⭐ 290

ENCODE ATAC-seq pipeline

most recent commit a year ago

Textpipe ⭐ 290

Textpipe: clean and extract metadata from text

total releases 39most recent commit 3 years ago

Cloudless ⭐ 290

Deep learning pipeline for orbital satellite data for detecting clouds

most recent commit 7 years ago

The 10 Most Depended On Python Pipeline Open Source Projects

Core ⭐ 166

The safe post-production pipeline - https://getavalon.github.io/2.0

dependent packages 1total releases 9latest release May 24, 2020most recent commit 3 years ago

Meltano ⭐ 1,460

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

dependent packages 1total releases 263latest release December 05, 2023most recent commit 3 months ago

Populse_mia ⭐ 10

Multiparametric Image Analysis

dependent packages 1total releases 14latest release July 01, 2022most recent commit 3 months ago

Vents ⭐ 30

Open source connections catalog, integrations, alerting, and notifications library.

dependent packages 1total releases 19latest release November 29, 2023most recent commit 3 months ago

Mara Pipelines ⭐ 2,053

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

dependent packages 1total releases 8latest release December 06, 2023most recent commit 4 months ago

Boxdetect ⭐ 59

BoxDetect is a Python package based on OpenCV which allows you to easily detect rectangular shapes like character or checkbox boxes on scanned forms.

dependent packages 1total releases 5latest release January 18, 2023most recent commit a year ago

Detect App ⭐ 7

Detect which app the python interpreter is running in.⚙️---------------------------- Supports Ansible AutoCAD Blender Calibre Cinema 4D Clarisse Flame Fusion Gaffer Gimp Houdini Inkscape Katana Krita Mari Marmoset 3ds Max Maya Modo MotionBuilder Natron Nuke RV Revit Shotgun Scribus Softimage SubstanceDesigner SubstancePainter Unreal

dependent packages 1total releases 4latest release October 11, 2023most recent commit 4 months ago

Tibanna ⭐ 68

Tibanna helps you run your genomic pipelines on Amazon cloud (AWS). It is used by the 4DN DCIC (4D Nucleome Data Coordination and Integration Center) to process data. Tibanna supports CWL/WDL (w/ docker), Snakemake (w/ conda) and custom Docker/shell command.

dependent packages 1total releases 280latest release October 06, 2023most recent commit 3 months ago

Igneous ⭐ 37

Scalable Neuroglancer compatible Downsampling, Meshing, Skeletonizing, Contrast Normalization, Transfers and more.

dependent packages 1total releases 56latest release November 21, 2023most recent commit 3 months ago

Augur ⭐ 217

Pipeline components for real-time phylodynamic analysis

dependent packages 1total releases 97latest release November 07, 2023most recent commit 3 months ago

The 10 Latest Releases In Python Pipeline Open Source Projects

Sdrf Pipelines ⭐ 12

A repository to convert SDRF proteomics files into pipelines config files

dependent packages 1total releases 22latest release September 17, 2023most recent commit 3 months ago

Ctapipe ⭐ 58

Low-level data processing pipeline software for the Cherenkov Telescope Array

dependent packages 11total releases 25latest release September 11, 2023most recent commit 3 months ago

Karton Config Extractor ⭐ 7

Static configuration extractor for the Karton framework

total releases 9latest release September 07, 2023most recent commit 3 months ago

Stimela Classic ⭐ 26

Containerized radio interferometry scripting framework -- NB: Classic version is no longer in active development, use stimela 2! See README for details.

dependent packages 4total releases 65latest release September 05, 2023most recent commit 6 months ago

Karton ⭐ 353

Distributed malware processing framework based on Python, Redis and S3.

dependent packages 11total releases 20latest release September 04, 2023most recent commit 3 months ago

Taskiq Pipelines ⭐ 6

Task pipelining for taskiq

total releases 4latest release September 03, 2023most recent commit 6 months ago

Monai Deploy App Sdk ⭐ 74

MONAI Deploy App SDK offers a framework and associated tools to design, develop and verify AI-driven applications in the healthcare imaging domain.

total releases 14latest release September 01, 2023most recent commit 5 months ago

Janis ⭐ 39

[Alpha] Janis: an open source tool to machine generate type-safe CWL and WDL workflows

dependent packages 7total releases 62latest release August 30, 2023most recent commit 8 months ago

Podpac ⭐ 43

Pipeline for Observational Data Processing Analysis and Collaboration

total releases 15latest release August 25, 2023most recent commit 4 months ago

Eventkit ⭐ 119

Event-driven data pipelines

dependent packages 6total releases 12latest release August 24, 2023most recent commit 4 months ago

Popular Data Processing Categories