Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Pipeline Big Data Open Source Projects

Open source projects categorized as Pipeline Big Data
Categories > Data Processing > Big Data
Categories > Data Processing > Pipeline
Beam ⭐ 7,355
Apache Beam is a unified programming model for Batch and Streaming data processing.
dependent packages 14total releases 568most recent commit 3 months ago
Pachyderm ⭐ 6,035
Data-Centric Pipelines and Data Versioning
dependent packages 1total releases 613most recent commit 3 months ago
Dataflowjavasdk ⭐ 853
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
dependent packages 14total releases 38most recent commit 3 years ago
Tez ⭐ 446
Apache Tez
most recent commit 3 months ago
Smooks ⭐ 377
Extensible data integration Java framework for building XML and non-XML fragment-based applications
dependent packages 14total releases 5most recent commit 3 months ago
Dataengineering Roadmap ⭐ 297
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
most recent commit 4 months ago
Big Data Rosetta Code ⭐ 283
Code snippets for solving common big data problems in various platforms. Inspired by Rosetta Code
most recent commit 5 months ago
Shifu ⭐ 235
An end-to-end machine learning and data mining framework on Hadoop
dependent packages 2total releases 9most recent commit a year ago
Mobydq ⭐ 175
🐳 Tool to automate data quality checks on data pipelines
most recent commit 2 years ago
Setl ⭐ 173
A simple Spark-powered ETL framework that just works 🍺
total releases 4most recent commit 4 months ago
Incubator Liminal ⭐ 131
Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.
total releases 28most recent commit 7 months ago
Crunch ⭐ 100
Mirror of Apache Crunch (Incubating)
most recent commit 3 years ago
Falcon ⭐ 95
Mirror of Apache Falcon
most recent commit 5 years ago
Ni ⭐ 81
Say "ni" to data of any size
most recent commit a year ago
Automaticanalysis ⭐ 70
Automatic Analysis (aa)
most recent commit 3 months ago
Banias ⭐ 37
Opinionated serverless event analytics pipeline
most recent commit 5 years ago
Nifi ⭐ 32
Deploy a secured, clustered, auto-scaling NiFi service in AWS.
most recent commit 2 years ago
Lidbox ⭐ 26
End-to-end spoken language identification out of the box. Rewrite in progress for first release (version 1).
total releases 6most recent commit 3 years ago
Hazelcast Jet Contrib ⭐ 19
Extension modules for Hazelcast Jet
most recent commit 2 years ago
Real Time Stock Analyzer ⭐ 16
Bigdata Pipeline
most recent commit 6 years ago
Snowplow Pipeline ⭐ 11
End-to-end Snowplow Analytics Pipeline for real time events
most recent commit 6 years ago
Lambdaconf 2017 Bigdata ⭐ 9
Materials for "Big Data Pipelines with Scala" Workshop at LambdaConf 2017
most recent commit 7 years ago
Airflow Data Pipeline ⭐ 7
Udacity Data Engineer Nanodegree - Airflow data pipeline
most recent commit 4 years ago
The 5 Most Depended On Pipeline Big Data Open Source Projects
Smooks ⭐ 377
Extensible data integration Java framework for building XML and non-XML fragment-based applications
dependent packages 14total releases 5latest release June 19, 2023most recent commit 3 months ago
Beam ⭐ 7,355
Apache Beam is a unified programming model for Batch and Streaming data processing.
dependent packages 14total releases 568latest release November 13, 2023most recent commit 3 months ago
Shifu ⭐ 235
An end-to-end machine learning and data mining framework on Hadoop
dependent packages 2total releases 9latest release April 03, 2019most recent commit a year ago
Pachyderm ⭐ 6,035
Data-Centric Pipelines and Data Versioning
dependent packages 1total releases 613latest release December 04, 2023most recent commit 3 months ago
The 8 Latest Releases In Pipeline Big Data Open Source Projects
Pachyderm ⭐ 6,035
Data-Centric Pipelines and Data Versioning
dependent packages 1total releases 613latest release December 04, 2023most recent commit 3 months ago
Beam ⭐ 7,355
Apache Beam is a unified programming model for Batch and Streaming data processing.
dependent packages 14total releases 568latest release November 13, 2023most recent commit 3 months ago
Smooks ⭐ 377
Extensible data integration Java framework for building XML and non-XML fragment-based applications
dependent packages 14total releases 5latest release June 19, 2023most recent commit 3 months ago
Setl ⭐ 173
A simple Spark-powered ETL framework that just works 🍺
total releases 4latest release August 21, 2020most recent commit 4 months ago
Shifu ⭐ 235
An end-to-end machine learning and data mining framework on Hadoop
dependent packages 2total releases 9latest release April 03, 2019most recent commit a year ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.