Awesome Open Source
Awesome Open Source
Combined Topics
big-data x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 1,898 Big Data Open Source Projects on Github

Categories > Data Processing > Big Data
Awesome Scalability ⭐ 35,830
The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
Spark ⭐ 30,849
Apache Spark - A unified analytics engine for large-scale data processing
Data Science Ipython Notebooks ⭐ 21,061
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Clickhouse ⭐ 19,118
ClickHouse® is a free analytics DBMS for big data
Flink ⭐ 17,154
Apache Flink
Tdengine ⭐ 16,020
An open-source big data platform designed and optimized for the Internet of Things (IoT).
Gun ⭐ 14,456
An open source cybersecurity protocol for syncing decentralized graph data.
Predictionio ⭐ 12,510
PredictionIO, a machine learning server for developers and ML engineers.
Presto ⭐ 12,428
The official home of the Presto distributed SQL query engine for big data
Cmak ⭐ 10,261
CMAK is a tool for managing Apache Kafka clusters
Awesome Bigdata ⭐ 10,187
A curated list of awesome big data frameworks, ressources and other awesomeness.
Bigdata Notes ⭐ 9,567
大数据入门指南 ⭐️
Cookbook ⭐ 9,275
The Data Engineering Cookbook
Vaex ⭐ 6,573
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualize and explore big tabular data at a billion rows per second 🚀
Cython ⭐ 6,396
The most widely used Python to C compiler
Storm ⭐ 6,276
Mirror of Apache Storm
Catboost ⭐ 6,060
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
H2o 3 ⭐ 5,534
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Zeppelin ⭐ 5,394
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Pachyderm ⭐ 5,214
Reproducible Data Science at Scale!
God Of Bigdata ⭐ 5,081
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
Couchdb ⭐ 5,058
Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability
Beam ⭐ 4,989
Apache Beam is a unified programming model for Batch and Streaming
Arkime ⭐ 4,908
Arkime (formerly Moloch) is an open source, large scale, full packet capturing, indexing, and database system.
Stream Framework ⭐ 4,555
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
Hazelcast ⭐ 4,541
Open-source distributed computation and storage platform
Trino ⭐ 4,094
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
Ignite ⭐ 3,975
Apache Ignite
Hive ⭐ 3,908
Apache Hive
Bigdl ⭐ 3,772
BigDL: Distributed Deep Learning Framework for Apache Spark
Delta ⭐ 3,645
An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.
Vespa ⭐ 3,625
The open big data serving engine. https://vespa.ai
Datahub ⭐ 3,560
A Metadata Platform for the Modern Data Stack
Crate ⭐ 3,183
CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time.
Vue Virtual Scroll List ⭐ 3,037
⚡️A vue component support big amount data list with high render performance and efficient.
Koalas ⭐ 2,907
Koalas: pandas API on Apache Spark
Cboard ⭐ 2,684
An easy to use, self-service open BI reporting and BI dashboard platform.
Calcite ⭐ 2,666
Apache Calcite
Dpark ⭐ 2,637
Python clone of Spark, a MapReduce alike framework in Python
Flinkx ⭐ 2,479
Based on Apache Flink. support data synchronization/integration and streaming SQL computation.
Nakedtensor ⭐ 2,471
Bare bone examples of machine learning in TensorFlow
Hudi ⭐ 2,267
Upserts, Deletes And Incremental Processing on Big Data.
Feast ⭐ 2,234
Feature Store for Machine Learning
Flume ⭐ 2,138
Mirror of Apache Flume
Volcano ⭐ 1,966
A Kubernetes Native Batch System (Project under CNCF)
Avro ⭐ 1,924
Apache Avro is a data serialization system.
Poli ⭐ 1,804
An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.
Spark ⭐ 1,677
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Gaffer ⭐ 1,634
A large-scale entity and relation database supporting aggregation of properties
Drill ⭐ 1,592
Apache Drill is a distributed MPP query layer for self describing data
Iotdb ⭐ 1,578
Apache IoTDB
Flinkstreamsql ⭐ 1,547
基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法
Poseidon ⭐ 1,543
A search engine which can hold 100 trillion lines of log data.
Griddb ⭐ 1,530
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
Genie ⭐ 1,515
Distributed Big Data Orchestration Service
Ambari ⭐ 1,514
Mirror of Apache Ambari
Just Dashboard ⭐ 1,489
📊 📋 Dashboards using YAML or JSON files
Kudu ⭐ 1,468
Mirror of Apache Kudu
Mysql_perf_analyzer ⭐ 1,420
MySQL performance monitoring and analysis.
Parquet Mr ⭐ 1,384
Apache Parquet
Bookkeeper ⭐ 1,380
Apache Bookkeeper
Mlsql ⭐ 1,345
The Programming Language Designed For Big Data and AI
Carbondata ⭐ 1,256
High performance data store solution
Spark Py Notebooks ⭐ 1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
My Journey In The Data Science World ⭐ 1,175
📢 Ready to learn or review your knowledge!
Bigdataguide ⭐ 1,164
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
Optimus ⭐ 1,106
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF and PySpark
Moosefs ⭐ 1,105
MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)
Spark Doc Zh ⭐ 1,101
Apache Spark 官方文档中文版
Datumbox Framework ⭐ 1,057
Datumbox is an open-source Machine Learning framework written in Java which allows the rapid development of Machine Learning and Statistical applications.
Bigdata Interview ⭐ 1,041
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Autocrawler ⭐ 1,038
Google, Naver multiprocess image web crawler (Selenium)
Egads ⭐ 1,007
A Java package to automatically detect anomalies in large scale time-series data
Tensorbase ⭐ 1,002
TensorBase is a new big data warehousing with modern efforts.
Traildb ⭐ 987
TrailDB is an efficient tool for storing and querying series of events
Dremio Oss ⭐ 955
Dremio - the missing link in modern data
Mobius ⭐ 930
C# and F# language binding and extensions to Apache Spark
Hazelcast Jet ⭐ 929
Distributed Stream and Batch Processing
Fooltrader ⭐ 912
a modular quant framework
Phoenix ⭐ 898
Mirror of Apache Phoenix
Panther ⭐ 894
[DEPRECATED] Detect threats with log data and improve cloud security posture
Parquet Format ⭐ 890
Apache Parquet
Accumulo ⭐ 886
Apache Accumulo
Dataflowjavasdk ⭐ 853
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Sqoop ⭐ 820
Mirror of Apache Sqoop
Autodl ⭐ 819
Automated Deep Learning without ANY human intervention. 1'st Solution for AutoDL [email protected]
Titanoboa ⭐ 817
Titanoboa makes complex workflows easy. It is a low-code workflow orchestration platform for JVM - distributed, highly scalable and fault tolerant.
Kube Batch ⭐ 790
A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC
Rakam Api ⭐ 772
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
Spark Movie Lens ⭐ 757
An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset
Gearpump ⭐ 747
Lightweight real-time big data streaming engine over Akka
Kafka Ui ⭐ 736
Open-Source Web GUI for Apache Kafka Management
Sciblog_support ⭐ 696
Support content for my blog
Samza ⭐ 695
Mirror of Apache Samza
Fluid ⭐ 678
Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)
Kafka Streams ⭐ 659
equivalent to kafka-streams 🐙 for nodejs ✨🐢🚀✨
Data Science Career ⭐ 655
Career Resources for Data Science, Machine Learning, Big Data and Business Analytics Career Repository
Cds ⭐ 651
Data syncing in golang for ClickHouse.
Sdc ⭐ 627
Intel® Scalable Dataframe Compiler for Pandas*
Oozie ⭐ 617
Mirror of Apache Oozie
1-100 of 1,898 projects

Related Projects

Python Big Data Projects (482)
Java Big Data Projects (446)
Spark Big Data Projects (393)
Machine Learning Big Data Projects (275)
Big Data Hadoop Projects (256)
Scala Big Data Projects (231)
Jupyter Notebook Big Data Projects (219)
Javascript Big Data Projects (173)
Scala Spark Big Data Projects (148)
Database Big Data Projects (122)
Spark Big Data Hadoop Projects (119)
Python Machine Learning Big Data Projects (117)
Python Spark Big Data Projects (116)
Kafka Big Data Projects (109)
Data Science Big Projects (4)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021