Awesome Open Source
Awesome Open Source
Combined Topics
data-processing x

The Top 690 Data Processing Open Source Projects

Categories > Data Processing > Data Processing
Bash Oneliner ⭐ 7,231
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
most recent commit 2 months ago
Miller ⭐ 5,811
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
total releases 64most recent commit a day ago
Awesome Web Scraping ⭐ 5,495
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 14 days ago
Deeplake ⭐ 5,038
Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets. Stream data real-time to PyTorch/TensorFlow. https://activeloop.ai
dependent packages 1total releases 149most recent commit 11 hours agopypi hub} Downloads
Dali ⭐ 4,119
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
most recent commit 13 hours ago
Dasel ⭐ 3,842
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
dependent packages 7total releases 80most recent commit 5 days ago
Texar ⭐ 2,008
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow
total releases 5most recent commit 2 years ago
Onyx ⭐ 1,995
Distributed, masterless, high performance, fault tolerant data processing
total releases 247most recent commit 3 years ago
Broadway ⭐ 1,954
Concurrent and multi-stage data ingestion and data processing with Elixir
dependent packages 22total releases 14most recent commit 6 days agohex broadway} Downloads
Pandera ⭐ 1,799
A light-weight, flexible, and expressive statistical data testing library
most recent commit 4 days ago
Dialogpt ⭐ 1,751
Large-scale pretraining for dialogue
most recent commit a month ago
Bonobo ⭐ 1,521
Extract Transform Load for Python 3.5+
dependent packages 7total releases 38most recent commit 4 months agopypi bonobo} Downloads
Cascalog ⭐ 1,362
Data processing on Hadoop without the hassle.
total releases 54most recent commit 3 years ago
Data Science On Gcp ⭐ 1,018
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
most recent commit 8 months ago
Satpy ⭐ 878
Python package for earth-observing satellite data processing
dependent packages 8total releases 77most recent commit 3 days agopypi satpy} Downloads
Dataflowjavasdk ⭐ 853
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
dependent packages 14total releases 38most recent commit 2 years ago
Texar Pytorch ⭐ 711
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
dependent packages 1total releases 5most recent commit 7 months agopypi texar-pytorch} Downloads
Qualitis ⭐ 527
Qualitis is a one-stop data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing. https://github.com/WeBankFinTech/Qualitis
most recent commit 6 days ago
Xidel ⭐ 517
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
most recent commit a month ago
Awesome Kafka ⭐ 513
A list about Apache Kafka
most recent commit 8 days ago
Traceml ⭐ 450
Engine for ML/Data tracking, visualization, dashboards, and model UI for Polyaxon
dependent packages 2total releases 18most recent commit 19 days agopypi traceml} Downloads
Collapse ⭐ 408
Advanced and Fast Data Transformation in R
dependent packages 8total releases 28most recent commit 24 days ago
Eternal ⭐ 372
👾~ music, eternal ~ 👾
most recent commit 9 months ago
Amadeus ⭐ 334
Harmonious distributed data analysis in Rust.
dependent packages 7total releases 25most recent commit a year agocargo amadeus-core} Downloads
Nonechucks ⭐ 315
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
total releases 18most recent commit a year ago
Numaflow ⭐ 301
Kubernetes-native platform to run massively parallel data/streaming jobs
most recent commit 10 hours ago
Covid Model ⭐ 296
most recent commit a year ago
Rapidtables ⭐ 284
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
dependent packages 7total releases 37most recent commit 3 years agopypi rapidtables} Downloads
Etl ⭐ 269
[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library
most recent commit 23 days ago
Redisgears ⭐ 265
Dynamic execution framework for your Redis data
most recent commit 7 days ago
Bytewax ⭐ 261
A Python framework for building highly scalable dataflows.
most recent commit 15 hours ago
Pulsar Flink ⭐ 255
Elastic data processing with Apache Pulsar and Apache Flink
total releases 71most recent commit 7 months ago
Pysparkling ⭐ 253
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
dependent packages 1total releases 68most recent commit 15 days agopypi pysparkling} Downloads
Machine Learning Notebooks ⭐ 241
Machine Learning notebooks for refreshing concepts.
most recent commit a year ago
Lithops ⭐ 240
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
dependent packages 1total releases 38most recent commit 4 days agopypi lithops} Downloads
50 Days Of Ml ⭐ 237
A day to day plan for this challenge (50 Days of Machine Learning) . Covers both theoretical and practical aspects
most recent commit 3 years ago
Scramjet ⭐ 233
A company that helps you bring data from environments together.
dependent packages 36total releases 177most recent commit 3 months agonpm scramjet} Downloads
Vaspy ⭐ 220
Manipulating VASP files with Python.
total releases 21most recent commit 5 months ago
Pxi ⭐ 217
🧚pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
total releases 30most recent commit 3 years ago
Godel ⭐ 204
Large-scale pretrained models for goal-directed dialog
most recent commit a month ago
Padasip ⭐ 201
Python Adaptive Signal Processing
dependent packages 1total releases 12most recent commit 4 months agopypi padasip} Downloads
Forte ⭐ 199
Forte is a flexible and powerful ML workflow builder. This is part of the CASL project: http://casl-project.ai/
dependent packages 12total releases 13most recent commit 23 days agopypi forte} Downloads
Batchflow ⭐ 183
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
total releases 7most recent commit 19 days ago
Convtools Ita ⭐ 176
convtools is a python library to declaratively define conversions for processing collections, doing complex aggregations and joins.
most recent commit a year ago
Dataflows ⭐ 171
DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.
dependent packages 11total releases 116most recent commit 3 months agopypi dataflows} Downloads
Sepal ⭐ 166
Geographical Data Processing in the Cloud
most recent commit 4 days ago
Mech ⭐ 155
🦾 Main repository for the Mech programming language. Start here!
total releases 5most recent commit a month ago
Salem ⭐ 153
Add geolocalised subsetting, masking, and plotting operations to xarray
most recent commit 3 months ago
Vue Datagrid ⭐ 152
Spreadsheet data grid component. Handles enormous data processing.
dependent packages 2total releases 67most recent commit 2 months agonpm @revolist/vue-datagrid} Downloads
Spatialspark ⭐ 141
Big Spatial Data Processing using Spark
most recent commit 6 years ago
Skills Ml ⭐ 136
Data Processing and Machine learning methods for the Open Skills Project
total releases 3most recent commit a year ago
Machine_learning_a Z ⭐ 130
Learning to create Machine Learning Algorithms
most recent commit a year ago
Cq ⭐ 124
Clojure Command-line Data Processor for JSON, YAML, EDN, XML and more
most recent commit 14 days ago
Data Processing Agreements ⭐ 116
Collection of Data Processing Agreement (DPA) and GDPR compliance resources
most recent commit 9 months ago
Splitpipeline ⭐ 112
Parallel Data Processing in PowerShell
total releases 28most recent commit 5 months ago
Sayn ⭐ 109
Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
total releases 27most recent commit 20 days ago
Shadow Experiments ⭐ 108
WIP. Do not use.
most recent commit 5 months ago
Distributed Dataset ⭐ 107
A distributed data processing framework in Haskell.
most recent commit 2 years ago
Dampr ⭐ 103
Python Data Processing library
total releases 9most recent commit 13 days ago
Blinkist M4a Downloader ⭐ 97
Grabs all of the audio files from all of the Blinkist books
most recent commit 2 months ago
Rsgislib ⭐ 96
Remote Sensing and GIS Software Library; python module tools for processing spatial data.
most recent commit 5 days ago
Biofsharp ⭐ 94
Open source bioinformatics and computational biology toolbox written in F#.
dependent packages 9total releases 18most recent commit 7 months agonuget BioFSharp} Downloads
Cotk ⭐ 93
Conversational Toolkit. An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
total releases 4most recent commit 2 years ago
Pulsar Spark ⭐ 93
When Apache Pulsar meets Apache Spark
total releases 3most recent commit 8 days ago
Dolphinnext ⭐ 92
A graphical user interface for distributed data processing of high throughput genomics
most recent commit 18 days ago
Libertem ⭐ 88
Open pixelated STEM framework
dependent packages 3total releases 19most recent commit 3 days agopypi libertem} Downloads
Text Dedup ⭐ 86
All-in-one text de-duplication
most recent commit 23 days ago
Breast Cancer Risk Prediction ⭐ 83
Classification of Breast Cancer diagnosis Using Support Vector Machines
most recent commit 2 years ago
Alfa ⭐ 80
♿️ Suite of open and standards-based tools for performing reliable accessibility conformance testing at scale
most recent commit 14 hours ago
Incubator Wayang ⭐ 74
Apache Wayang(incubating) is the first cross-platform data processing system.
most recent commit 2 months ago
2019 Electronic Design Competition ⭐ 70
【电赛】2019 全国大学生电子设计竞赛 （F题）纸张数量检测装置 （基于STM32F407 & FDC2214 & USART HMI）
most recent commit a year ago
Deep Learn Oil ⭐ 68
Deep learning tools for predicting oil well data
most recent commit a year ago
Ijcai18 Mama Ads Competition ⭐ 68
IJCAI-18 阿里妈妈搜索广告转化预测初赛方案
most recent commit 5 years ago
Ipython Notebooks ⭐ 62
most recent commit 4 years ago
Cbrain ⭐ 60
CBRAIN is a flexible Ruby on Rails framework for accessing and processing of large data on high-performance computing infrastructures.
most recent commit 6 days ago
Vip ⭐ 57
VIP is a python package/library for angular, reference star and spectral differential imaging for exoplanet/disk detection through high-contrast imaging.
total releases 31most recent commit 8 hours ago
Machine Learning For Solar Energy Prediction ⭐ 57
Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
most recent commit 3 years ago
Smartcitiesdata ⭐ 56
The core micro services of UrbanOS as an umbrella project with component documentation
most recent commit 8 hours ago
Tubes ⭐ 55
A series of tubes.
dependent packages 2total releases 5most recent commit 6 months agopypi Tubes} Downloads
Processor ⭐ 54
Ontology-driven Linked Data processor and server for SPARQL backends. Apache License.
dependent packages 1total releases 32most recent commit 24 days ago
Mdsplus ⭐ 54
The MDSplus data management system
total releases 65most recent commit 5 days ago
Prosto ⭐ 53
Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
total releases 5most recent commit a year ago
Data_processing_course ⭐ 52
Some class materials for a data processing course using PySpark
most recent commit 8 days ago
Housing Insights ⭐ 52
Bringing open data to affordable housing decision makers in Washington DC. A D3/Javascript based website to visualize data related to affordable housing in Washington DC. Data processing with Python.
most recent commit 3 years ago
Blaze ⭐ 52
A blazing fast exporter for your Elasticsearch data.
most recent commit 2 years ago
Tqdj ⭐ 51
A progress bar that plays lofi music
total releases 2most recent commit 2 years ago
Storm Docker ⭐ 50
Docker image packaging for Apache Storm
most recent commit 4 months ago
Lrs3 For Speech Separation ⭐ 49
Multi-modal speech separation task data generation script on LRS3 data set.
most recent commit 6 months ago
Zenquery ⭐ 47
Enterprise backend as a service
most recent commit 4 years ago
Rsna Intracranial ⭐ 47
most recent commit 3 years ago
Itertable ⭐ 46
⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLSX, XML, and JSON.
dependent packages 1total releases 3most recent commit 5 months agopypi itertable} Downloads
Jqnpm ⭐ 42
A package manager built for the command-line JSON processor jq.
most recent commit a year ago
Pyseis ⭐ 41
Pure python seismic data processing
most recent commit a year ago
Data Science Using Python University Course Module ⭐ 40
“Data science” is just about as broad of a term as they come. It may be easiest to describe what it is by listing its more concrete components: Data exploration & analysis. Included here: Pandas; NumPy; SciPy; a helping hand from Python's Standard Library.
most recent commit 3 years ago
Five Dollar Genome Analysis Pipeline ⭐ 40
Workflows used for WGS data processing -- replaced by https://github.com/gatk-workflows/gatk4-genome-pro
most recent commit 3 years ago
Datacook ⭐ 38
Machine Learning and Data Analysis in JavaScript.
total releases 56most recent commit a month ago
Executiongraph ⭐ 36
Fast Generic Execution Graph/Network
most recent commit 9 months ago
Pygaps ⭐ 36
A framework for processing adsorption data and isotherm fitting
total releases 25most recent commit 3 months ago
User2018 ⭐ 35
Slides and code for the 2018 useR! tutorial "Recipes for Data Processing"
most recent commit 4 years ago
Tutorials ⭐ 35
Tutorials for spatial data processing and analysis in R and Python
most recent commit 4 years ago
1-100 of 690 projects
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,114,612)
Python  (860,459)
Java  (402,966)
Php  (292,746)
Typescript  (281,047)
C Plus Plus  (251,243)
Ruby  (223,115)
C  (189,468)
Shell  (178,484)
C Sharp  (173,233)
Golang  (167,331)
Swift  (67,673)
Rust  (64,694)
R  (60,818)
Dart  (58,623)
Objective C  (57,044)
Kotlin  (50,945)
Lua  (36,944)
Matlab  (32,297)
Perl  (30,697)
Language  (29,924)
Scala  (29,110)
Haskell  (20,749)
Powershell  (19,905)
Clojure  (19,387)
Bash  (19,012)
Coffeescript  (17,177)
Elixir  (17,147)
Assembly  (15,188)
Julia  (13,615)
Processing  (13,290)
Basic  (10,688)
Emacs Lisp  (10,669)
Flow  (10,517)
Solidity  (9,511)
Erlang  (9,088)
Groovy  (8,508)
Scheme  (8,422)
Ocaml  (6,870)
Pascal  (6,688)
Hack  (6,547)
Lisp  (6,346)
Fortran  (5,726)
Elm  (5,537)
Common Lisp  (5,421)
Visual Basic  (4,960)
Dlang  (4,780)
Shell Script  (4,761)
Python Library  (4,491)
Crystal  (4,419)
Privacy | About | Terms | Follow Us On Twitter | Top Projects

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.