Project Name	Stars	Repos Using This	Packages Using This	Most Recent Commit	Total Releases	Latest Release	Open Issues	License	Language
Airbyte	12,918		11	3 months ago	311	December 08, 2023	5,111	other	Python
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Goodreads_etl_pipeline	593			4 years ago				mit	Python
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Datajoint Python	158	12	26	4 months ago	66	September 21, 2022	143	lgpl-2.1	Python
Relational data pipelines for the science lab
Aws Pdf Textract Pipeline	148			3 months ago			5	mit	TypeScript
:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
Rds Snapshot Export To S3 Pipeline	102			10 months ago			5	mit-0	TypeScript
RDS Snapshot Export to S3 Pipeline
Scrapy S3pipeline	66	1		2 years ago	8	January 31, 2021	1	mit	Python
Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.
Automating Livestream Video Monitoring	34			8 months ago			2	mit-0	Jupyter Notebook
This repo presents a demo application for realtime livestream video quality monitoring using AWS serverless and AI/ML services.
Nifi	32			2 years ago					HCL
Deploy a secured, clustered, auto-scaling NiFi service in AWS.
Spark Movies Etl	21			8 months ago			2		Python
Spark data pipeline that ingests and transforms movie ratings data.
Saisoku	21			4 years ago				apache-2.0	Python
Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Alternatives To Saisoku

Select To Compare

Airbyte ⭐ 12,918

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

dependent packages 11total releases 311most recent commit 3 months ago

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

most recent commit 4 years ago

Datajoint Python ⭐ 158

Relational data pipelines for the science lab

dependent packages 26total releases 66most recent commit 4 months ago

Aws Pdf Textract Pipeline ⭐ 148

:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

most recent commit 3 months ago

Rds Snapshot Export To S3 Pipeline ⭐ 102

RDS Snapshot Export to S3 Pipeline

most recent commit 10 months ago

Scrapy S3pipeline ⭐ 66

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

total releases 8most recent commit 2 years ago

Automating Livestream Video Monitoring ⭐ 34

This repo presents a demo application for realtime livestream video quality monitoring using AWS serverless and AI/ML services.

most recent commit 8 months ago

Nifi ⭐ 32

Deploy a secured, clustered, auto-scaling NiFi service in AWS.

most recent commit 2 years ago

Spark Movies Etl ⭐ 21

Spark data pipeline that ingests and transforms movie ratings data.

most recent commit 8 months ago

Saisoku ⭐ 21

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

most recent commit 4 years ago

Suggest An Alternative To saisoku

Alternative Project Comparisons

Saisoku vs Airbyte

Saisoku vs Goodreads_etl_pipeline

Saisoku vs Datajoint Python

Saisoku vs Aws Pdf Textract Pipeline

Saisoku vs Rds Snapshot Export To S3 Pipeline

Saisoku vs Scrapy S3pipeline

Saisoku vs Automating Livestream Video Monitoring

Saisoku vs Nifi

Saisoku vs Spark Movies Etl

Popular Pipeline Projects

Nushell ⭐ 28,304

A new type of shell

dependent packages 82total releases 77latest release November 20, 2023most recent commit 3 months ago

Jina ⭐ 19,573

☁️ Build multimodal AI applications with cloud-native stack

dependent packages 16total releases 2,453latest release December 01, 2023most recent commit 3 months ago

Luigi ⭐ 17,046

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

dependent packages 76total releases 80latest release October 05, 2023most recent commit 3 months ago

Vector ⭐ 16,326

A high-performance observability data pipeline.

total releases 3latest release March 13, 2021most recent commit 10 days ago

Argo Cd ⭐ 15,229

Declarative Continuous Deployment for Kubernetes

dependent packages 65total releases 359latest release December 01, 2023most recent commit 3 months ago

Popular S3 Projects

Minio ⭐ 42,715

The Object Store for AI Data Infrastructure

dependent packages 129total releases 377latest release December 01, 2023most recent commit 3 months ago

Rclone ⭐ 42,258

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

dependent packages 43total releases 268latest release November 26, 2023most recent commit 3 months ago

Seaweedfs ⭐ 20,994

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

dependent packages 2total releases 296latest release April 24, 2021most recent commit a day ago

Transfer.sh ⭐ 14,640

Easy and fast file sharing from the command-line.

dependent packages 2total releases 27latest release December 04, 2023most recent commit 4 months ago

Siyuan ⭐ 14,236

A privacy-first, self-hosted, fully open source personal knowledge management software, written in typescript and golang.

total releases 1latest release July 07, 2022most recent commit 3 months ago

Popular Data Processing Categories