Awesome Open Source
Awesome Open Source
Combined Topics
data-processing
x
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
The Top 317 Data Processing Open Source Projects on Github
Categories
>
Data Processing
>
Data Processing
Miller
⭐
4,383
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
Awesome Web Scraping
⭐
4,262
List of libraries, tools and APIs for web scraping and data processing.
Hub
⭐
3,529
Fastest dataset optimization and management for machine and deep learning. Stream data real-time & version-control it. http://activeloop.ai
Dali
⭐
3,467
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
Texar
⭐
2,008
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow
Broadway
⭐
1,568
Concurrent and multi-stage data ingestion and data processing with Elixir
Dasel
⭐
1,557
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
Bash Oneliner
⭐
1,489
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Bonobo
⭐
1,408
Extract Transform Load for Python 3.5+
Dialogpt
⭐
1,393
Large-scale pretraining for dialogue
Data Science On Gcp
⭐
911
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Dataflowjavasdk
⭐
853
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Pandera
⭐
705
A light-weight, flexible, and expressive pandas data validation library
Texar Pytorch
⭐
682
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Awesome Kafka
⭐
436
A list about Apache Kafka
Xidel
⭐
367
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Amadeus
⭐
334
Harmonious distributed data analysis in Rust.
Nonechucks
⭐
315
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
Rapidtables
⭐
284
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
Machine Learning Notebooks
⭐
241
Machine Learning notebooks for refreshing concepts.
Collapse
⭐
240
Advanced and Fast Data Transformation in R
Pysparkling
⭐
234
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
Pxi
⭐
217
🧚pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
Pulsar Flink
⭐
213
Elastic data processing with Apache Pulsar and Apache Flink
50 Days Of Ml
⭐
199
A day to day plan for this challenge (50 Days of Machine Learning) . Covers both theoretical and practical aspects
Convtools
⭐
175
convtools is a python library to declaratively define conversions for processing collections, doing complex aggregations and joins.
Lithops
⭐
156
An open source framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud.
Vaspy
⭐
153
Manipulating VASP files with Python.
Padasip
⭐
149
Python Adaptive Signal Processing
Forte
⭐
142
Forte is a flexible and powerful NLP builder FOR TExt. This is part of the CASL project: http://casl-project.ai/
Machine_learning_a Z
⭐
130
Learning to create Machine Learning Algorithms
Distributed Dataset
⭐
107
A distributed data processing framework in Haskell.
Data Processing Agreements
⭐
101
Collection of Data Processing Agreement (DPA) and GDPR compliance resources
Cotk
⭐
93
Conversational Toolkit. An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
Breast Cancer Risk Prediction
⭐
83
Classification of Breast Cancer diagnosis Using Support Vector Machines
Mech
⭐
82
Main repository for the Mech programming language. Start here! 🦾
Biofsharp
⭐
81
Open source bioinformatics and computational biology toolbox written in F#.
Libertem
⭐
73
Open pixelated STEM framework
2019 Electronic Design Competition
⭐
70
【电赛】2019 全国大学生电子设计竞赛 （F题）纸张数量检测装置 （基于STM32F407 & FDC2214 & USART HMI）
Pulsar Spark
⭐
69
When Apache Pulsar meets Apache Spark
Ijcai18 Mama Ads Competition
⭐
68
IJCAI-18 阿里妈妈搜索广告转化预测初赛方案
Blinkist M4a Downloader
⭐
66
Grabs all of the audio files from all of the Blinkist books
Alfa
⭐
66
♿️ Suite of open and standards-based tools for performing reliable accessibility conformance testing at scale
Machine Learning For Solar Energy Prediction
⭐
57
Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
Vip
⭐
52
VIP is a python package/library for angular, reference star and spectral differential imaging for exoplanet/disk detection through high-contrast imaging.
Cbrain
⭐
52
CBRAIN is a flexible Ruby on Rails framework for accessing and processing of large data on high-performance computing infrastructures.
Blaze
⭐
52
A blazing fast exporter for your Elasticsearch data.
Prosto
⭐
48
Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
Deep Learn Oil
⭐
48
Deep learning tools for predicting oil well data
Storm Docker
⭐
48
Docker image packaging for Apache Storm
Mdsplus
⭐
48
The MDSplus data management system
Zenquery
⭐
47
Enterprise backend as a service
Data_processing_course
⭐
45
Some class materials for a data processing course using PySpark
Lrs3 For Speech Separation
⭐
44
Multi-modal speech separation task data generation script on LRS3 data set.
Jqnpm
⭐
42
A package manager built for the command-line JSON processor jq.
Processor
⭐
41
Ontology-driven Linked Data processor and server for SPARQL backends. Apache License.
Smartcitiesdata
⭐
40
Smart Cities Data platform umbrella project and component documentation
Data Science Using Python University Course Module
⭐
40
“Data science” is just about as broad of a term as they come. It may be easiest to describe what it is by listing its more concrete components: Data exploration & analysis. Included here: Pandas; NumPy; SciPy; a helping hand from Python's Standard Library.
Rheem
⭐
37
Rheem - a cross-platform data processing system
Itertable
⭐
33
⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLS, XML, and JSON.
P4
⭐
30
P4: Portable Parallel Processing Pipeline
Ms2txt
⭐
28
Little utility to decode Metastock files and write them in text format
Stargate
⭐
26
An Apache Pulsar client written in Elixir
Processor
⭐
26
A simple and lightweight JavaScript data processing tool. Live demo:
Pygaps
⭐
26
A framework for processing adsorption data and isotherm fitting
Meta Schema
⭐
25
Little DSL to make data processing sane with clojure.spec and spec-tools
Executiongraph
⭐
24
Fast Generic Execution Graph/Network
Zenaton Python
⭐
24
🐍 Python library to run and orchestrate background jobs with Zenaton Workflow Engine
Bakeryjs
⭐
23
Dataprocessing framework for JavaScript 🎂
Bdit_data Sources
⭐
21
Data sources used by the Big Data Innovation Team
Machine Learning Data Pipeline
⭐
20
Pipeline module for parallel real-time data processing for machine learning models development and production purposes.
Data Processing And Visualization
⭐
19
This document forms the basis of several workshops/talks that get into everyday programming with R, but also includes mirrored code in Python as Jupyter notebooks.
Ibm Cloud Functions Data Processing Message Hub
⭐
19
Create a serverless, event-driven application with Apache OpenWhisk on IBM Cloud Functions that executes code in response to messages or to handle streams of data records from Apache Kafka or IBM Message Hub.
Rec Core
⭐
19
Data pipelining service
Restaurant Finder Featurereviews
⭐
19
Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).
Speech Recognition
⭐
18
End-to-end Automatic Speech Recognition for Madarian and English in Tensorflow
Bonobo Sqlalchemy
⭐
18
PREVIEW - SQL databases in Bonobo, using sqlalchemy
Pax
⭐
16
The XENON1T raw data processor [deprecated]
Prairie
⭐
16
A visual programming environment for Python
Sparklanes
⭐
16
A lightweight data processing framework for Apache Spark
Ecg_analysis
⭐
16
Ds Project Maker
⭐
14
Template repository for initializing data science projects. Designed for student project work on the Make School Data Science track.
Automated Data Preprocessing
⭐
14
A command-line utility program for automating the trivial, frequently occurring data preparation tasks: missing value interpolation, outlier removal, and encoding categorical variables.
Brepnet
⭐
13
BRepNet: A topological message passing system for solid models
Glide
⭐
12
Easy ETL
Pulserl
⭐
12
Apache Pulsar client library for Erlang/Elixir
Baleen3
⭐
12
Baleen 3 is a data processing tool based on the Annot8 framework
Wind
⭐
11
The Flow-based Programming Language
Abacuza
⭐
11
Easing your on-premise Data Processing
Tumor Cell Segmentation
⭐
10
tumor cell segmentation by inception-v3 and FCN model
Meds Processor
⭐
10
Learn C# and .NET Core by building a scraper, downloader and parser for Croatia's Health Insurance Fund primary and supplementary drugs list.
Rpi
⭐
10
RPJiOS: RPJ's RPi OS, a sensor data platform for the Raspberry Pi built with python2.7 and redis.
Ibm Cloud Functions Message Hub Trigger
⭐
10
IBM Cloud Functions building block - Message Hub Trigger - This project provides a starting point for handling events from Message Hub with IBM Cloud Functions powered by Apache OpenWhisk.
Nodalflow
⭐
10
A PHP Nodal (net)WorkFlow
Cs229 Machine Learning Solar Energy Predictions
⭐
10
Predicting solar energy using machine learning (LSTM, PCA, boosting). This is our CS 229 project from autumn 2017. Report and poster are included.
Tablecloth.time
⭐
10
Tools for the processing and manipulation of time-series data in Clojure.
Pdb4amber
⭐
10
PDB analyzer and fixer for Amber simulations
Pywikimm
⭐
9
Collects a multimodal dataset of Wikipedia articles and their images
Nwpc Data
⭐
9
A data tool for operational systems in NWPC/CMA.
Pulsar Hub
⭐
9
The canonical source of StreamNative Hub.
1-100 of 317 projects
Next >
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.
