Awesome Open Source
Awesome Open Source
Combined Topics
big-data x

The Top 2,674 Big Data Open Source Projects

Categories > Data Processing > Big Data
Awesome Scalability ⭐ 39,079
The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
most recent commit 6 days ago
Spark ⭐ 33,280
Apache Spark - A unified analytics engine for large-scale data processing
dependent packages 882total releases 46most recent commit 10 hours ago
Clickhouse ⭐ 24,376
ClickHouse® is a free analytics DBMS for big data
total releases 699most recent commit 10 hours ago
Data Science Ipython Notebooks ⭐ 22,662
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
most recent commit 3 months ago
Flink ⭐ 19,241
Apache Flink
dependent packages 284total releases 62most recent commit 14 hours ago
Public Apis ⭐ 18,625
📚 A public list of APIs from round the web.
most recent commit 9 days ago
Tdengine ⭐ 18,575
An open-source time-series database with high-performance, scalability and SQL support. It can be widely used in IoT, Connected Vehicles, DevOps, Energy, Finance and other fields.
dependent packages 1total releases 12most recent commit 10 hours agonpm td2.0-connector} Downloads
Shardingsphere ⭐ 16,219
Ecosystem to transform any database into a distributed database system, and enhance it with sharding, elastic scaling, encryption features & more
dependent packages 8total releases 7most recent commit 11 hours ago
Gun ⭐ 16,096
An open source cybersecurity protocol for syncing decentralized graph data.
dependent packages 111total releases 234most recent commit 7 days agonpm gun} Downloads
Presto ⭐ 13,618
The official home of the Presto distributed SQL query engine for big data
dependent packages 11total releases 272most recent commit 18 hours ago
Predictionio ⭐ 12,510
PredictionIO, a machine learning server for developers and ML engineers.
dependent packages 8total releases 5most recent commit a year ago
Bigdata Notes ⭐ 11,183
大数据入门指南 ⭐️
most recent commit 5 months ago
Awesome Bigdata ⭐ 10,864
A curated list of awesome big data frameworks, ressources and other awesomeness.
most recent commit a month ago
Cmak ⭐ 10,754
CMAK is a tool for managing Apache Kafka clusters
most recent commit 2 months ago
Cookbook ⭐ 9,863
The Data Engineering Cookbook
most recent commit 6 months ago
Vaex ⭐ 7,131
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
dependent packages 15total releases 89most recent commit 5 days agopypi vaex-core} Downloads
Cython ⭐ 7,116
The most widely used Python to C compiler
dependent packages 1,822total releases 115most recent commit 4 days agopypi Cython} Downloads
God Of Bigdata ⭐ 7,010
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive.
most recent commit 20 hours ago
Catboost ⭐ 6,616
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
dependent packages 5total releases 59most recent commit a day ago
Storm ⭐ 6,366
Mirror of Apache Storm
total releases 7most recent commit 16 days ago
H2o 3 ⭐ 5,874
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
dependent packages 30total releases 227most recent commit 16 hours ago
Zeppelin ⭐ 5,730
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
dependent packages 23total releases 2most recent commit 3 days agonpm zeppelin-vis} Downloads
Datahub ⭐ 5,723
The Metadata Platform for the Modern Data Stack
dependent packages 4total releases 210most recent commit a day agopypi acryl-datahub} Downloads
Trino ⭐ 5,658
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
dependent packages 9total releases 51most recent commit 11 hours ago
Beam ⭐ 5,647
Apache Beam is a unified programming model for Batch and Streaming data processing.
dependent packages 11total releases 527most recent commit 13 hours ago
Pachyderm ⭐ 5,530
Data-Centric Pipelines and Data Versioning
dependent packages 1total releases 185most recent commit a day ago
Couchdb ⭐ 5,365
Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability
most recent commit 18 hours ago
Arkime ⭐ 5,213
Arkime (formerly Moloch) is an open source, large scale, full packet capturing, indexing, and database system.
most recent commit a day ago
Hazelcast ⭐ 4,905
Open-source distributed computation and storage platform
dependent packages 361total releases 237most recent commit 2 days ago
Doris ⭐ 4,678
Apache Doris is an MPP-based interactive SQL data warehousing for reporting and analysis.
most recent commit 9 hours ago
Stream Framework ⭐ 4,555
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
total releases 98most recent commit a year ago
Delta ⭐ 4,476
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python.
dependent packages 10total releases 15most recent commit 8 days ago
Hive ⭐ 4,350
Apache Hive
most recent commit 17 hours ago
Ignite ⭐ 4,195
Apache Ignite
most recent commit 15 hours ago
Vespa ⭐ 3,973
The open big data serving engine. https://vespa.ai
dependent packages 25total releases 533most recent commit 13 hours ago
Crate ⭐ 3,434
CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time.
dependent packages 1total releases 13most recent commit a day ago
Synapseml ⭐ 3,348
Simple and Distributed Machine Learning
dependent packages 1total releases 3most recent commit a day agopypi synapseml} Downloads
Feast ⭐ 3,332
Feature Store for Machine Learning
dependent packages 19total releases 91most recent commit 17 hours agopypi feast} Downloads
Hudi ⭐ 3,279
Upserts, Deletes And Incremental Processing on Big Data.
dependent packages 25total releases 11most recent commit 9 hours ago
Vue Virtual Scroll List ⭐ 3,272
⚡️A vue component support big amount data list with high render performance and efficient.
dependent packages 37total releases 81most recent commit 5 months agonpm vue-virtual-scroll-list} Downloads
Calcite ⭐ 3,174
Apache Calcite
dependent packages 94total releases 1,708most recent commit 2 days ago
Kafka Ui ⭐ 3,086
Open-Source Web GUI for Apache Kafka Management
most recent commit a day ago
Chunjun ⭐ 3,068
Based on Apache Flink. Support data synchronization/integration.
most recent commit 6 days ago
Koalas ⭐ 2,999
Koalas: pandas API on Apache Spark
dependent packages 12total releases 47most recent commit 8 months agopypi koalas} Downloads
Blaze ⭐ 2,949
NumPy and Pandas interface to Big Data
dependent packages 1total releases 1most recent commit 2 years ago
Cboard ⭐ 2,764
An easy to use, self-service open BI reporting and BI dashboard platform.
most recent commit a year ago
Starrocks ⭐ 2,762
StarRocks is a next-gen sub-second MPP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query.
most recent commit 10 hours ago
Dpark ⭐ 2,637
Python clone of Spark, a MapReduce alike framework in Python
total releases 19most recent commit 2 years ago
Volcano ⭐ 2,488
A Cloud Native Batch System (Project under CNCF)
dependent packages 21total releases 105most recent commit 14 hours ago
Nakedtensor ⭐ 2,471
Bare bone examples of machine learning in TensorFlow
most recent commit 5 years ago
Flume ⭐ 2,275
Mirror of Apache Flume
most recent commit 18 days ago
H2o 2 ⭐ 2,242
Please visit https://github.com/h2oai/h2o-3 for latest H2O
most recent commit 2 years ago
Avro ⭐ 2,172
Apache Avro is a data serialization system.
dependent packages 1,173total releases 148most recent commit a day ago
Arrow Datafusion ⭐ 2,171
Apache Arrow DataFusion SQL Query Engine
dependent packages 27total releases 57most recent commit 13 hours agocargo datafusion} Downloads
Iotdb ⭐ 2,035
Apache IoTDB
dependent packages 7total releases 15most recent commit 14 hours ago
Incubator Hugegraph ⭐ 2,012
A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)
dependent packages 1total releases 5most recent commit 2 days ago
Sql Generator ⭐ 1,923
🔨 用 JSON 来生成结构化的 SQL 语句，基于 Vue3 + TypeScript + Vite + Ant Design + MonacoEditor 实现，项目简单（重逻辑轻页面）、适合练手~
most recent commit a month ago
Poli ⭐ 1,868
An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.
most recent commit 4 months ago
Root ⭐ 1,837
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
dependent packages 19total releases 14most recent commit 12 hours ago
Fastjson2 ⭐ 1,809
🚄 FASTJSON2是FASTJSON项目的重要升级，目标是为下一个十年提供一个高性能的JSON库
dependent packages 2,326total releases 267most recent commit 12 hours ago
Spark ⭐ 1,778
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
dependent packages 15total releases 20most recent commit a month agonuget Microsoft.Spark} Downloads
Flinkstreamsql ⭐ 1,735
基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法
most recent commit 5 months ago
Griddb ⭐ 1,693
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
total releases 7most recent commit 2 months ago
Drill ⭐ 1,688
Apache Drill is a distributed MPP query layer for self describing data
dependent packages 11total releases 20most recent commit 3 days ago
Streamx ⭐ 1,677
Make stream processing easier! Flink & Spark development scaffold, The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.
total releases 4most recent commit 19 hours ago
Gaffer ⭐ 1,658
A large-scale entity and relation database supporting aggregation of properties
dependent packages 10total releases 90most recent commit 15 days ago
Parquet Mr ⭐ 1,588
Apache Parquet
dependent packages 142total releases 13most recent commit 3 days ago
Kudu ⭐ 1,586
Mirror of Apache Kudu
most recent commit a day ago
Bookkeeper ⭐ 1,584
Apache Bookkeeper
dependent packages 24total releases 38most recent commit 18 hours ago
Bigdataguide ⭐ 1,572
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
most recent commit 11 days ago
Genie ⭐ 1,571
Distributed Big Data Orchestration Service
dependent packages 8total releases 208most recent commit 2 months ago
Ambari ⭐ 1,564
Mirror of Apache Ambari
total releases 2most recent commit 8 months ago
Byzer Lang ⭐ 1,564
Byzer (former MLSQL): A low-code open-source programming language for data pipeline, analytics and AI.
dependent packages 4total releases 2most recent commit 24 days ago
Poseidon ⭐ 1,543
A search engine which can hold 100 trillion lines of log data.
most recent commit 5 years ago
Just Dashboard ⭐ 1,489
📊 📋 Dashboards using YAML or JSON files
dependent packages 1total releases 29most recent commit a year agonpm just-dashboard} Downloads
Vaquarkhan ⭐ 1,435
most recent commit 20 days ago
Mysql_perf_analyzer ⭐ 1,420
MySQL performance monitoring and analysis.
most recent commit a year ago
Carbondata ⭐ 1,322
High performance data store solution
most recent commit 2 days ago
Bigdata Interview ⭐ 1,237
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop
most recent commit 10 months ago
Optimus ⭐ 1,227
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
total releases 27most recent commit 11 days ago
Spark Py Notebooks ⭐ 1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit 5 years ago
Tensorbase ⭐ 1,217
TensorBase is a new big data warehousing with modern efforts.
most recent commit 2 months ago
Moosefs ⭐ 1,171
MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)
most recent commit 7 months ago
Spark Doc Zh ⭐ 1,101
Apache Spark 官方文档中文版
most recent commit 2 years ago
Fluid ⭐ 1,087
Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)
dependent packages 1total releases 146most recent commit 19 hours ago
Dremio Oss ⭐ 1,084
Dremio - the missing link in modern data
most recent commit 5 days ago
Openmetadata ⭐ 1,084
Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
dependent packages 1total releases 111most recent commit a day agopypi openmetadata-ingestion} Downloads
Datumbox Framework ⭐ 1,057
Datumbox is an open-source Machine Learning framework written in Java which allows the rapid development of Machine Learning and Statistical applications.
dependent packages 4total releases 4most recent commit 2 years ago
Egads ⭐ 1,052
A Java package to automatically detect anomalies in large scale time-series data
total releases 2most recent commit 6 months ago
Autocrawler ⭐ 1,038
Google, Naver multiprocess image web crawler (Selenium)
most recent commit a year ago
Utils4s ⭐ 1,033
scala、spark使用过程中，各种测试用例以及相关资料整理
most recent commit 3 years ago
Parquet Format ⭐ 1,019
Apache Parquet
dependent packages 12total releases 8most recent commit 3 months ago
Traildb ⭐ 987
TrailDB is an efficient tool for storing and querying series of events
most recent commit 2 years ago
Hazelcast Jet ⭐ 975
Distributed Stream and Batch Processing
most recent commit 3 months ago
Kube Batch ⭐ 969
A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC
dependent packages 27total releases 5most recent commit 12 hours ago
Autodl ⭐ 944
Automated Deep Learning without ANY human intervention. 1'st Solution for AutoDL [email protected]
total releases 2most recent commit 5 months ago
Phoenix ⭐ 935
Mirror of Apache Phoenix
most recent commit 9 days ago
Mobius ⭐ 930
C# and F# language binding and extensions to Apache Spark
total releases 22most recent commit a year ago
Adam ⭐ 927
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
dependent packages 17total releases 14most recent commit 22 days ago
Accumulo ⭐ 922
Apache Accumulo
most recent commit 2 days ago
1-100 of 2,674 projects
Related Awesome Lists
Python Big Data Projects (588)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 71
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 343
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 115
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,060,211)
Python  (807,641)
Java  (388,609)
Php  (286,034)
Typescript  (246,628)
C Plus Plus  (238,913)
Ruby  (221,842)
C  (182,072)
Shell  (171,290)
C Sharp  (164,223)
Golang  (158,346)
Swift  (65,252)
R  (57,963)
Rust  (57,638)
Objective C  (57,193)
Dart  (52,664)
Kotlin  (46,490)
Lua  (33,910)
Matlab  (31,164)
Perl  (30,540)
Language  (30,058)
Scala  (28,670)
Haskell  (20,376)
Clojure  (19,140)
Powershell  (19,032)
Bash  (18,677)
Coffeescript  (17,289)
Elixir  (16,663)
Assembly  (14,686)
Processing  (13,308)
Julia  (12,753)
Basic  (10,734)
Flow  (10,548)
Emacs Lisp  (10,418)
Erlang  (9,065)
Groovy  (8,445)
Scheme  (8,438)
Solidity  (7,976)
Ocaml  (6,705)
Pascal  (6,438)
Hack  (6,396)
Lisp  (6,341)
Fortran  (5,542)
Elm  (5,484)
Common Lisp  (5,300)
Dlang  (4,745)
Shell Script  (4,654)
Python Library  (4,455)
Crystal  (4,389)
Scripting  (4,205)
Top Projects
Freecodecamp ⭐ 348,310
996.icu ⭐ 262,763
Free Programming Books ⭐ 237,644
Coding Interview University ⭐ 222,708
Awesome ⭐ 206,565
Public Apis ⭐ 198,632
Developer Roadmap ⭐ 198,514
Vue ⭐ 197,304
React ⭐ 190,704
System Design Primer ⭐ 184,236
Tensorflow ⭐ 166,096
Bootstrap ⭐ 158,261
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 151,618
Ohmyzsh ⭐ 146,966
Build Your Own X ⭐ 145,201
Javascript Algorithms ⭐ 143,983
Flutter ⭐ 142,298
Python ⭐ 139,362
Gitignore ⭐ 134,582
Linux ⭐ 134,002
Vscode ⭐ 133,521
Awesome Python ⭐ 132,049
Javascript ⭐ 124,360
Javaguide ⭐ 124,065
Python 100 Days ⭐ 120,229
Computer Science ⭐ 118,096
Youtube Dl ⭐ 111,107
Fucking Algorithm ⭐ 107,431
The Art Of Command Line ⭐ 106,741
React Native ⭐ 103,564
Electron ⭐ 102,334
D3 ⭐ 101,735
Go ⭐ 101,197
30 Seconds Of Code ⭐ 96,728
Create React App ⭐ 95,709
Axios ⭐ 94,209
Free Programming Books Zh_cn ⭐ 93,295
Awesome Selfhosted ⭐ 93,129
Kubernetes ⭐ 89,794
Next.js ⭐ 88,965
Node ⭐ 88,517
Terminal ⭐ 83,811
Deno ⭐ 83,497
Three.js ⭐ 83,238
Awesome Go ⭐ 83,181
Angular ⭐ 82,417
Typescript ⭐ 81,908
Ant Design ⭐ 80,905
Material Ui ⭐ 79,421
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.