Awesome Open Source
Awesome Open Source
Combined Topics
data-processing x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 34 Data Processing Open Source Projects

Categories > Data Processing > Data Processing
Awesome Web Scraping ⭐ 4,100
List of libraries, tools and APIs for web scraping and data processing.
Dali ⭐ 3,218
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
Miller ⭐ 2,710
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
Texar ⭐ 2,130
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Bonobo ⭐ 1,414
Extract Transform Load for Python 3.5+
Bash Oneliner ⭐ 1,383
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Broadway ⭐ 1,345
Concurrent and multi-stage data ingestion and data processing with Elixir
Dialogpt ⭐ 1,215
Large-scale pretraining for dialogue
Data Science On Gcp ⭐ 875
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Dataflowjavasdk ⭐ 853
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Texar Pytorch ⭐ 649
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Pandera ⭐ 542
A light-weight, flexible, and expressive pandas data validation library
Awesome Kafka ⭐ 409
A list about Apache Kafka
Xidel ⭐ 346
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Eternal ⭐ 328
👾~ music, eternal ~ 👾
Amadeus ⭐ 309
Harmonious distributed data analysis in Rust.
Nonechucks ⭐ 307
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
Rapidtables ⭐ 291
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
Pxi ⭐ 248
🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
Pysparkling ⭐ 231
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
Machine Learning Notebooks ⭐ 226
Machine Learning notebooks for refreshing concepts.
Vaspy ⭐ 189
Manipulating VASP files with Python.
Collapse ⭐ 187
Advanced and Fast Data Transformation in R
Padasip ⭐ 139
Python Adaptive Signal Processing
Pulsar Flink ⭐ 134
Elastic data processing with Apache Pulsar and Apache Flink
Data Processing Agreements ⭐ 112
Collection of Data Processing Agreement (DPA) and GDPR compliance resources
Distributed Dataset ⭐ 111
A distributed data processing framework in Haskell.
Machine Learning For Solar Energy Prediction ⭐ 99
Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
Forte ⭐ 98
Forte is a flexible and powerful NLP builder FOR TExt. This is part of the CASL project: http://casl-project.ai/
Pulsar Spark ⭐ 57
When Apache Pulsar meets Apache Spark
2019 Electronic Design Competition ⭐ 52
【电赛】2019 全国大学生电子设计竞赛 （F题）纸张数量检测装置 （基于STM32F407 & FDC2214 & USART HMI）
Cbrain ⭐ 51
CBRAIN is a flexible Ruby on Rails framework for accessing and processing of large data on high-performance computing infrastructures.
Mdsplus ⭐ 47
The MDSplus data management system
Tdm ⭐ 33
R package for normalizing RNA-seq data to make them comparable to microarray data.
1-34 of 34 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210