Awesome Open Source
Awesome Open Source
Combined Topics
big-data x

The Top 2,661 Big Data Open Source Projects

Categories > Data Processing > Big Data
Awesome Scalability ⭐ 38,595
The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
most recent commit 2 days ago
Spark ⭐ 32,967
Apache Spark - A unified analytics engine for large-scale data processing
dependent packages 882total releases 46most recent commit 13 hours ago
Clickhouse ⭐ 23,846
ClickHouse® is a free analytics DBMS for big data
total releases 699most recent commit 13 hours ago
Data Science Ipython Notebooks ⭐ 22,662
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
most recent commit 2 months ago
Flink ⭐ 18,992
Apache Flink
dependent packages 284total releases 62most recent commit 15 hours ago
Tdengine ⭐ 18,386
An open-source time-series database with high-performance, scalability and SQL support. It can be widely used in IoT, Connected Vehicles, DevOps, Energy, Finance and other fields.
dependent packages 1total releases 12most recent commit 13 hours agonpm td2.0-connector} Downloads
Public Apis ⭐ 18,382
📚 A public list of APIs from round the web.
most recent commit 13 days ago
Gun ⭐ 15,970
An open source cybersecurity protocol for syncing decentralized graph data.
dependent packages 111total releases 234most recent commit 3 days agonpm gun} Downloads
Shardingsphere ⭐ 15,965
Ecosystem to transform any database into a distributed database system, and enhance it with sharding, elastic scaling, encryption features & more
dependent packages 8total releases 7most recent commit 15 hours ago
Presto ⭐ 13,503
The official home of the Presto distributed SQL query engine for big data
dependent packages 11total releases 269most recent commit 12 hours ago
Predictionio ⭐ 12,510
PredictionIO, a machine learning server for developers and ML engineers.
dependent packages 8total releases 5most recent commit a year ago
Bigdata Notes ⭐ 11,183
大数据入门指南 ⭐️
most recent commit 4 months ago
Awesome Bigdata ⭐ 10,862
A curated list of awesome big data frameworks, ressources and other awesomeness.
most recent commit 9 days ago
Awesome Bigdata ⭐ 10,849
A curated list of awesome big data frameworks, ressources and other awesomeness.
most recent commit 9 days ago
Cmak ⭐ 10,754
CMAK is a tool for managing Apache Kafka clusters
most recent commit a month ago
Cookbook ⭐ 9,863
The Data Engineering Cookbook
most recent commit 5 months ago
Vaex ⭐ 7,083
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
dependent packages 15total releases 60most recent commit a day agopypi vaex} Downloads
Cython ⭐ 7,036
The most widely used Python to C compiler
dependent packages 1,822total releases 115most recent commit 3 days agopypi Cython} Downloads
God Of Bigdata ⭐ 6,749
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive.
most recent commit 19 days ago
Catboost ⭐ 6,532
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
dependent packages 5total releases 59most recent commit 18 hours ago
Storm ⭐ 6,356
Mirror of Apache Storm
total releases 7most recent commit a day ago
H2o 3 ⭐ 5,830
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
dependent packages 30total releases 226most recent commit 15 hours ago
Zeppelin ⭐ 5,662
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
dependent packages 23total releases 2most recent commit 15 days agonpm zeppelin-vis} Downloads
Datahub ⭐ 5,543
The Metadata Platform for the Modern Data Stack
dependent packages 4total releases 192most recent commit 13 hours agopypi acryl-datahub} Downloads
Beam ⭐ 5,530
Apache Beam is a unified programming model for Batch and Streaming data processing.
dependent packages 11total releases 527most recent commit 13 hours ago
Pachyderm ⭐ 5,501
The Complete MLOps Stack
dependent packages 1total releases 180most recent commit 14 hours ago
Trino ⭐ 5,461
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
dependent packages 9total releases 51most recent commit 12 hours ago
Couchdb ⭐ 5,307
Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability
most recent commit 8 days ago
Arkime ⭐ 5,165
Arkime (formerly Moloch) is an open source, large scale, full packet capturing, indexing, and database system.
most recent commit 2 days ago
Hazelcast ⭐ 4,863
Open-source distributed computation and storage platform
dependent packages 360total releases 236most recent commit 2 days ago
Stream Framework ⭐ 4,555
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
total releases 98most recent commit 10 months ago
Delta ⭐ 4,376
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python.
dependent packages 10total releases 15most recent commit 2 days ago
Hive ⭐ 4,292
Apache Hive
most recent commit 15 hours ago
Ignite ⭐ 4,161
Apache Ignite
most recent commit a day ago
Vespa ⭐ 3,942
The open big data serving engine. https://vespa.ai
dependent packages 25total releases 533most recent commit 13 hours ago
Crate ⭐ 3,404
CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time.
dependent packages 1total releases 13most recent commit 3 days ago
Synapseml ⭐ 3,301
Simple and Distributed Machine Learning
dependent packages 1total releases 3most recent commit 3 days agopypi synapseml} Downloads
Vue Virtual Scroll List ⭐ 3,272
⚡️A vue component support big amount data list with high render performance and efficient.
dependent packages 37total releases 81most recent commit 4 months agonpm vue-virtual-scroll-list} Downloads
Feast ⭐ 3,214
Feature Store for Machine Learning
dependent packages 19total releases 89most recent commit 13 hours agopypi feast} Downloads
Hudi ⭐ 3,157
Upserts, Deletes And Incremental Processing on Big Data.
dependent packages 25total releases 11most recent commit 13 hours ago
Calcite ⭐ 3,116
Apache Calcite
dependent packages 94total releases 1,708most recent commit 18 hours ago
Chunjun ⭐ 3,002
Based on Apache Flink. Support data synchronization/integration.
most recent commit a day ago
Koalas ⭐ 2,999
Koalas: pandas API on Apache Spark
dependent packages 12total releases 47most recent commit 7 months agopypi koalas} Downloads
Blaze ⭐ 2,949
NumPy and Pandas interface to Big Data
dependent packages 1total releases 1most recent commit 2 years ago
Kafka Ui ⭐ 2,872
Open-Source Web GUI for Apache Kafka Management
most recent commit 12 hours ago
Flinkx ⭐ 2,774
Based on Apache Flink. Support data synchronization/integration.
most recent commit 3 months ago
Cboard ⭐ 2,764
An easy to use, self-service open BI reporting and BI dashboard platform.
most recent commit a year ago
Dpark ⭐ 2,637
Python clone of Spark, a MapReduce alike framework in Python
total releases 19most recent commit a year ago
Starrocks ⭐ 2,594
StarRocks is a next-gen sub-second MPP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query.
most recent commit 13 hours ago
Nakedtensor ⭐ 2,471
Bare bone examples of machine learning in TensorFlow
most recent commit 5 years ago
Volcano ⭐ 2,404
A Cloud Native Batch System (Project under CNCF)
dependent packages 20total releases 103most recent commit 2 days ago
Flume ⭐ 2,262
Mirror of Apache Flume
most recent commit 17 days ago
H2o 2 ⭐ 2,242
Please visit https://github.com/h2oai/h2o-3 for latest H2O
most recent commit 2 years ago
Avro ⭐ 2,133
Apache Avro is a data serialization system.
dependent packages 1,165total releases 148most recent commit 6 days ago
Arrow Datafusion ⭐ 2,036
Apache Arrow DataFusion SQL Query Engine
dependent packages 23total releases 56most recent commit 16 hours agocargo datafusion} Downloads
Iotdb ⭐ 1,978
Apache IoTDB
dependent packages 7total releases 15most recent commit 14 hours ago
Incubator Hugegraph ⭐ 1,975
A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)
dependent packages 1total releases 5most recent commit 10 days ago
Sql Generator ⭐ 1,923
🔨 用 JSON 来生成结构化的 SQL 语句，基于 Vue3 + TypeScript + Vite + Ant Design + MonacoEditor 实现，项目简单（重逻辑轻页面）、适合练手~
most recent commit 5 days ago
Poli ⭐ 1,868
An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.
most recent commit 3 months ago
Root ⭐ 1,809
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
dependent packages 19total releases 13most recent commit 21 hours ago
Spark ⭐ 1,778
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
dependent packages 15total releases 19most recent commit 3 days agonuget Microsoft.Spark} Downloads
Flinkstreamsql ⭐ 1,735
基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法
most recent commit 3 months ago
Griddb ⭐ 1,693
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
total releases 7most recent commit a month ago
Drill ⭐ 1,679
Apache Drill is a distributed MPP query layer for self describing data
dependent packages 11total releases 20most recent commit 14 hours ago
Gaffer ⭐ 1,655
A large-scale entity and relation database supporting aggregation of properties
dependent packages 10total releases 90most recent commit a day ago
Genie ⭐ 1,571
Distributed Big Data Orchestration Service
dependent packages 8total releases 208most recent commit 18 days ago
Kudu ⭐ 1,568
Mirror of Apache Kudu
most recent commit 2 days ago
Streamx ⭐ 1,566
Make stream processing easier! Flink & Spark development scaffold, The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.
total releases 4most recent commit 2 days ago
Ambari ⭐ 1,564
Mirror of Apache Ambari
total releases 2most recent commit 6 months ago
Byzer Lang ⭐ 1,560
Byzer (former MLSQL): A low-code open-source programming language for data pipeline, analytics and AI.
dependent packages 4total releases 2most recent commit 10 days ago
Fastjson2 ⭐ 1,560
🚄 FASTJSON2是FASTJSON项目的重要升级，目标是为下一个十年提供一个高性能的JSON库
dependent packages 2,322total releases 236most recent commit 13 hours ago
Bookkeeper ⭐ 1,555
Apache Bookkeeper
dependent packages 24total releases 38most recent commit 3 days ago
Poseidon ⭐ 1,543
A search engine which can hold 100 trillion lines of log data.
most recent commit 5 years ago
Parquet Mr ⭐ 1,533
Apache Parquet
dependent packages 141total releases 12most recent commit 9 days ago
Bigdataguide ⭐ 1,514
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
most recent commit a month ago
Just Dashboard ⭐ 1,489
📊 📋 Dashboards using YAML or JSON files
dependent packages 1total releases 29most recent commit a year agonpm just-dashboard} Downloads
Mysql_perf_analyzer ⭐ 1,420
MySQL performance monitoring and analysis.
most recent commit a year ago
Carbondata ⭐ 1,317
High performance data store solution
most recent commit a day ago
Vaquarkhan ⭐ 1,247
most recent commit a year ago
Bigdata Interview ⭐ 1,237
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop
most recent commit 9 months ago
Spark Py Notebooks ⭐ 1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit 5 years ago
Tensorbase ⭐ 1,217
TensorBase is a new big data warehousing with modern efforts.
most recent commit 18 days ago
Optimus ⭐ 1,213
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
total releases 26most recent commit 24 days ago
Moosefs ⭐ 1,171
MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)
most recent commit 6 months ago
Spark Doc Zh ⭐ 1,101
Apache Spark 官方文档中文版
most recent commit a year ago
Fluid ⭐ 1,072
Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)
dependent packages 1total releases 146most recent commit a day ago
Dremio Oss ⭐ 1,068
Dremio - the missing link in modern data
most recent commit 14 hours ago
Datumbox Framework ⭐ 1,057
Datumbox is an open-source Machine Learning framework written in Java which allows the rapid development of Machine Learning and Statistical applications.
dependent packages 4total releases 4most recent commit 2 years ago
Egads ⭐ 1,052
A Java package to automatically detect anomalies in large scale time-series data
total releases 2most recent commit 5 months ago
Autocrawler ⭐ 1,038
Google, Naver multiprocess image web crawler (Selenium)
most recent commit 10 months ago
Utils4s ⭐ 1,033
scala、spark使用过程中，各种测试用例以及相关资料整理
most recent commit 3 years ago
Parquet Format ⭐ 1,019
Apache Parquet
dependent packages 12total releases 8most recent commit 2 months ago
Traildb ⭐ 987
TrailDB is an efficient tool for storing and querying series of events
most recent commit 2 years ago
Openmetadata ⭐ 977
Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
dependent packages 1total releases 13most recent commit 13 hours agopypi openmetadata-ingestion-core} Downloads
Hazelcast Jet ⭐ 975
Distributed Stream and Batch Processing
most recent commit a month ago
Autodl ⭐ 944
Automated Deep Learning without ANY human intervention. 1'st Solution for AutoDL [email protected]
total releases 2most recent commit 4 months ago
Kube Batch ⭐ 935
A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC
dependent packages 27total releases 5most recent commit 7 months ago
Phoenix ⭐ 931
Mirror of Apache Phoenix
most recent commit 9 days ago
Mobius ⭐ 930
C# and F# language binding and extensions to Apache Spark
total releases 22most recent commit a year ago
Adam ⭐ 925
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
dependent packages 17total releases 14most recent commit 5 days ago
1-100 of 2,661 projects
Related Awesome Lists
Python Big Data Projects (588)
Spark Big Data Projects (570)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,049,307)
Python  (796,774)
Java  (385,554)
Php  (284,659)
Typescript  (240,091)
C Plus Plus  (236,336)
Ruby  (221,670)
C  (180,445)
Shell  (169,696)
C Sharp  (162,146)
Golang  (156,215)
Swift  (64,723)
R  (57,307)
Objective C  (57,160)
Rust  (56,175)
Dart  (51,536)
Kotlin  (45,555)
Lua  (33,316)
Matlab  (30,910)
Perl  (30,480)
Language  (30,081)
Scala  (28,595)
Haskell  (20,266)
Clojure  (19,073)
Powershell  (18,808)
Bash  (18,609)
Coffeescript  (17,327)
Elixir  (16,573)
Assembly  (14,535)
Processing  (13,303)
Julia  (12,548)
Basic  (10,762)
Flow  (10,527)
Emacs Lisp  (10,374)
Erlang  (9,056)
Scheme  (8,438)
Groovy  (8,433)
Solidity  (7,630)
Ocaml  (6,669)
Pascal  (6,384)
Hack  (6,366)
Lisp  (6,340)
Fortran  (5,485)
Elm  (5,478)
Common Lisp  (5,278)
Dlang  (4,745)
Shell Script  (4,633)
Python Library  (4,449)
Crystal  (4,375)
Scripting  (4,203)
Top Projects
Freecodecamp ⭐ 346,483
996.icu ⭐ 262,320
Free Programming Books ⭐ 233,869
Coding Interview University ⭐ 220,499
Awesome ⭐ 202,760
Vue ⭐ 196,227
Developer Roadmap ⭐ 195,459
Public Apis ⭐ 194,093
React ⭐ 188,524
System Design Primer ⭐ 180,789
Tensorflow ⭐ 165,263
Bootstrap ⭐ 157,530
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,579
Build Your Own X ⭐ 142,363
Javascript Algorithms ⭐ 141,923
Flutter ⭐ 140,992
Python ⭐ 136,366
Gitignore ⭐ 133,520
Linux ⭐ 132,253
Vscode ⭐ 132,135
Awesome Python ⭐ 127,528
Javascript ⭐ 123,498
Javaguide ⭐ 121,716
Python 100 Days ⭐ 117,585
Computer Science ⭐ 113,544
Youtube Dl ⭐ 110,286
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,897
Electron ⭐ 101,831
D3 ⭐ 100,901
Go ⭐ 99,557
30 Seconds Of Code ⭐ 95,885
Create React App ⭐ 95,200
Axios ⭐ 93,611
Awesome Selfhosted ⭐ 89,230
Kubernetes ⭐ 88,745
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,866
Next.js ⭐ 87,217
Terminal ⭐ 83,297
Deno ⭐ 82,633
Three.js ⭐ 82,255
Angular ⭐ 81,550
Awesome Go ⭐ 81,188
Typescript ⭐ 80,934
Ant Design ⭐ 80,442
Material Ui ⭐ 78,665
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.