Awesome Open Source
Awesome Open Source
Combined Topics
data-processing
x
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
The Top 333 Data Processing Open Source Projects on Github
Categories
>
Data Processing
>
Data Processing
Miller
⭐
4,922
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
Awesome Web Scraping
⭐
4,539
List of libraries, tools and APIs for web scraping and data processing.
Hub
⭐
4,220
Dataset format for AI. Build, manage, & visualize datasets for deep learning. Stream data real-time to PyTorch/TensorFlow & version-control it. https://activeloop.ai
Dali
⭐
3,680
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
Texar
⭐
2,008
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow
Dasel
⭐
1,745
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
Broadway
⭐
1,705
Concurrent and multi-stage data ingestion and data processing with Elixir
Bash Oneliner
⭐
1,489
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Bonobo
⭐
1,472
Extract Transform Load for Python 3.5+
Dialogpt
⭐
1,430
Large-scale pretraining for dialogue
Data Science On Gcp
⭐
987
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Pandera
⭐
938
A light-weight, flexible, and expressive data validation library for dataframes
Dataflowjavasdk
⭐
853
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Texar Pytorch
⭐
698
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Awesome Kafka
⭐
462
A list about Apache Kafka
Xidel
⭐
427
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Amadeus
⭐
334
Harmonious distributed data analysis in Rust.
Nonechucks
⭐
315
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
Collapse
⭐
289
Advanced and Fast Data Transformation in R
Rapidtables
⭐
284
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
Machine Learning Notebooks
⭐
241
Machine Learning notebooks for refreshing concepts.
Pulsar Flink
⭐
241
Elastic data processing with Apache Pulsar and Apache Flink
50 Days Of Ml
⭐
237
A day to day plan for this challenge (50 Days of Machine Learning) . Covers both theoretical and practical aspects
Pysparkling
⭐
234
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
Pxi
⭐
217
🧚pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
Lithops
⭐
183
An open source framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud.
Convtools Ita
⭐
176
convtools is a python library to declaratively define conversions for processing collections, doing complex aggregations and joins.
Padasip
⭐
168
Python Adaptive Signal Processing
Forte
⭐
154
Forte is a flexible and powerful NLP builder FOR TExt. This is part of the CASL project: http://casl-project.ai/
Vaspy
⭐
153
Manipulating VASP files with Python.
Machine_learning_a Z
⭐
130
Learning to create Machine Learning Algorithms
Distributed Dataset
⭐
107
A distributed data processing framework in Haskell.
Data Processing Agreements
⭐
101
Collection of Data Processing Agreement (DPA) and GDPR compliance resources
Cotk
⭐
93
Conversational Toolkit. An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
Mech
⭐
91
Main repository for the Mech programming language. Start here! 🦾
Biofsharp
⭐
89
Open source bioinformatics and computational biology toolbox written in F#.
Breast Cancer Risk Prediction
⭐
83
Classification of Breast Cancer diagnosis Using Support Vector Machines
Libertem
⭐
78
Open pixelated STEM framework
Pulsar Spark
⭐
76
When Apache Pulsar meets Apache Spark
Alfa
⭐
72
♿️ Suite of open and standards-based tools for performing reliable accessibility conformance testing at scale
2019 Electronic Design Competition
⭐
70
【电赛】2019 全国大学生电子设计竞赛 （F题）纸张数量检测装置 （基于STM32F407 & FDC2214 & USART HMI）
Ijcai18 Mama Ads Competition
⭐
68
IJCAI-18 阿里妈妈搜索广告转化预测初赛方案
Deep Learn Oil
⭐
68
Deep learning tools for predicting oil well data
Blinkist M4a Downloader
⭐
66
Grabs all of the audio files from all of the Blinkist books
Machine Learning For Solar Energy Prediction
⭐
57
Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
Cbrain
⭐
56
CBRAIN is a flexible Ruby on Rails framework for accessing and processing of large data on high-performance computing infrastructures.
Vip
⭐
54
VIP is a python package/library for angular, reference star and spectral differential imaging for exoplanet/disk detection through high-contrast imaging.
Prosto
⭐
53
Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
Blaze
⭐
52
A blazing fast exporter for your Elasticsearch data.
Storm Docker
⭐
50
Docker image packaging for Apache Storm
Mdsplus
⭐
50
The MDSplus data management system
Data_processing_course
⭐
48
Some class materials for a data processing course using PySpark
Zenquery
⭐
47
Enterprise backend as a service
Cq
⭐
46
Clojure Command-line Data Processor for JSON, YAML, EDN, XML and more
Smartcitiesdata
⭐
46
Smart Cities Data platform umbrella project and component documentation
Lrs3 For Speech Separation
⭐
44
Multi-modal speech separation task data generation script on LRS3 data set.
Incubator Wayang
⭐
42
Apache Wayang(incubating) is the first cross-platform data processing system.
Jqnpm
⭐
42
A package manager built for the command-line JSON processor jq.
Processor
⭐
41
Ontology-driven Linked Data processor and server for SPARQL backends. Apache License.
Itertable
⭐
41
⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLSX, XML, and JSON.
Data Science Using Python University Course Module
⭐
40
“Data science” is just about as broad of a term as they come. It may be easiest to describe what it is by listing its more concrete components: Data exploration & analysis. Included here: Pandas; NumPy; SciPy; a helping hand from Python's Standard Library.
Tdm
⭐
34
R package for normalizing RNA-seq data to make them comparable to microarray data.
Executiongraph
⭐
34
Fast Generic Execution Graph/Network
Parallel Corpora Tools
⭐
32
Tools for filtering and cleaning parallel and monolingual corpora for machine translation and other natural language processing tasks.
P4
⭐
30
P4: Portable Parallel Processing Pipeline
Ms2txt
⭐
28
Little utility to decode Metastock files and write them in text format
Processor
⭐
26
A simple and lightweight JavaScript data processing tool. Live demo:
Pygaps
⭐
26
A framework for processing adsorption data and isotherm fitting
Stargate
⭐
26
An Apache Pulsar client written in Elixir
Meta Schema
⭐
25
Little DSL to make data processing sane with clojure.spec and spec-tools
Zenaton Python
⭐
24
🐍 Python library to run and orchestrate background jobs with Zenaton Workflow Engine
Bakeryjs
⭐
23
Dataprocessing framework for JavaScript 🎂
Brepnet
⭐
21
BRepNet: A topological message passing system for solid models
Bdit_data Sources
⭐
21
Data sources used by the Big Data Innovation Team
Machine Learning Data Pipeline
⭐
20
Pipeline module for parallel real-time data processing for machine learning models development and production purposes.
Rec Core
⭐
19
Data pipelining service
Restaurant Finder Featurereviews
⭐
19
Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).
Ibm Cloud Functions Data Processing Message Hub
⭐
19
Create a serverless, event-driven application with Apache OpenWhisk on IBM Cloud Functions that executes code in response to messages or to handle streams of data records from Apache Kafka or IBM Message Hub.
Data Processing And Visualization
⭐
19
This document forms the basis of several workshops/talks that get into everyday programming with R, but also includes mirrored code in Python as Jupyter notebooks.
Speech Recognition
⭐
18
End-to-end Automatic Speech Recognition for Madarian and English in Tensorflow
Bonobo Sqlalchemy
⭐
18
PREVIEW - SQL databases in Bonobo, using sqlalchemy
Sparklanes
⭐
16
A lightweight data processing framework for Apache Spark
Ecg_analysis
⭐
16
Prairie
⭐
16
A visual programming environment for Python
Pax
⭐
16
The XENON1T raw data processor [deprecated]
Abacuza
⭐
16
Easing your on-premise Data Processing
Anatomy Of System Engineering
⭐
15
System Engineering Memory Map
Baleen3
⭐
15
Baleen 3 is a data processing tool based on the Annot8 framework
Automated Data Preprocessing
⭐
14
A command-line utility program for automating the trivial, frequently occurring data preparation tasks: missing value interpolation, outlier removal, and encoding categorical variables.
Ds Project Maker
⭐
14
Template repository for initializing data science projects. Designed for student project work on the Make School Data Science track.
Glide
⭐
12
Easy ETL
Pulserl
⭐
12
Apache Pulsar client library for Erlang/Elixir
Task Farmer
⭐
12
A simple multi-core task scheduler that works well with promises. Great for doing parallel data processing.
Tablecloth.time
⭐
11
Tools for the processing and manipulation of time-series data in Clojure.
Wind
⭐
11
The Flow-based Programming Language
Tasrif
⭐
11
Tasrif is a python library for processing of wearable data from fitness trackers and wearable health devices
Meds Processor
⭐
10
Learn C# and .NET Core by building a scraper, downloader and parser for Croatia's Health Insurance Fund primary and supplementary drugs list.
Tumor Cell Segmentation
⭐
10
tumor cell segmentation by inception-v3 and FCN model
Cdp Backend
⭐
10
Data storage utilities and processing pipelines used by CDP instances.
Thepipe
⭐
10
A simplistic, general purpose pipeline framework.
1-100 of 333 projects
Next >
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
Privacy policy
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.
