Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Data Processing Open Source Projects

Open source projects categorized as Data Processing
Categories > Data Processing > Data Processing
Miller ⭐ 8,397
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
total releases 65most recent commit 3 months ago
Bash Oneliner ⭐ 7,946
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
most recent commit 5 months ago
Awesome Web Scraping ⭐ 6,060
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 5 months ago
Dasel ⭐ 4,845
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
dependent packages 14total releases 97most recent commit 11 days ago
Dali ⭐ 4,770
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
dependent packages 4total releases 19most recent commit 3 months agopypi nvidia-dali-cuda110} Downloads
Pandera ⭐ 2,807
A light-weight, flexible, and expressive statistical data testing library
dependent packages 97total releases 79most recent commit 3 months agopypi pandera} Downloads
Dialogpt ⭐ 2,314
Large-scale pretraining for dialogue
most recent commit 2 years ago
Broadway ⭐ 2,233
Concurrent and multi-stage data ingestion and data processing with Elixir
dependent packages 29total releases 18most recent commit 3 months agohex broadway} Downloads
Texar ⭐ 2,008
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow
total releases 5most recent commit 4 years ago
Bonobo ⭐ 1,548
Extract Transform Load for Python 3.5+
dependent packages 7total releases 38most recent commit a year agopypi bonobo} Downloads
Cascalog ⭐ 1,378
Data processing on Hadoop without the hassle.
total releases 54most recent commit a year ago
Data Science On Gcp ⭐ 1,249
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
most recent commit 9 months ago
Satpy ⭐ 980
Python package for earth-observing satellite data processing
dependent packages 14total releases 91most recent commit 3 months agopypi satpy} Downloads
Bytewax ⭐ 957
Python Stream Processing
dependent packages 3total releases 34most recent commit 3 months agopypi bytewax} Downloads
Numaflow ⭐ 866
Kubernetes-native platform to run massively parallel data/streaming jobs
dependent packages 1total releases 37most recent commit 3 months ago
Dataflowjavasdk ⭐ 853
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
dependent packages 14total releases 38most recent commit 3 years ago
Godel ⭐ 831
Large-scale pretrained models for goal-directed dialog
most recent commit 4 months ago
Texar Pytorch ⭐ 711
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
dependent packages 3total releases 5most recent commit 2 years agopypi texar-pytorch} Downloads
Hstream ⭐ 668
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
total releases 12most recent commit 3 months ago
Qualitis ⭐ 631
Qualitis is a one-stop data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing. https://github.com/WeBankFinTech/Qualitis
most recent commit 3 months ago
Xidel ⭐ 611
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
most recent commit 5 months ago
Collapse ⭐ 556
Advanced and Fast Data Transformation in R
dependent packages 19total releases 38most recent commit 3 months ago
Awesome Kafka ⭐ 549
A list about Apache Kafka
most recent commit 7 months ago
The 10 Most Depended On Data Processing Open Source Projects
Pandera ⭐ 2,807
A light-weight, flexible, and expressive statistical data testing library
dependent packages 97total releases 79latest release December 08, 2023most recent commit 3 months agopypi pandera} Downloads
Catmandu ⭐ 170
Catmandu - a data processing toolkit
dependent packages 86total releases 150latest release November 06, 2023most recent commit 5 months ago
Broadway ⭐ 2,233
Concurrent and multi-stage data ingestion and data processing with Elixir
dependent packages 29total releases 18latest release April 22, 2023most recent commit 3 months agohex broadway} Downloads
Collapse ⭐ 556
Advanced and Fast Data Transformation in R
dependent packages 19total releases 38latest release December 07, 2023most recent commit 3 months ago
Etl ⭐ 327
PHP - ETL (Extract Transform Load) data processing library
dependent packages 15total releases 16latest release November 30, 2023most recent commit 3 months agopackagist flow-php/etl} Downloads
Satpy ⭐ 980
Python package for earth-observing satellite data processing
dependent packages 14total releases 91latest release November 29, 2023most recent commit 3 months agopypi satpy} Downloads
Incubator Wayang ⭐ 162
Apache Wayang(incubating) is the first cross-platform data processing system.
dependent packages 13total releases 2latest release September 03, 2023most recent commit 3 months ago
Forte ⭐ 215
Forte is a flexible and powerful ML workflow builder. This is part of the CASL project: http://casl-project.ai/
dependent packages 12total releases 14latest release June 29, 2022most recent commit a year agopypi forte} Downloads
The 10 Latest Releases In Data Processing Open Source Projects
Fondant ⭐ 293
Production-ready data processing made easy and shareable
total releases 24latest release December 12, 2023most recent commit 3 months ago
Haupt ⭐ 451
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
dependent packages 1total releases 86latest release December 10, 2023most recent commit 9 days agopypi haupt} Downloads
Pandera ⭐ 2,807
A light-weight, flexible, and expressive statistical data testing library
dependent packages 97total releases 79latest release December 08, 2023most recent commit 3 months agopypi pandera} Downloads
Collapse ⭐ 556
Advanced and Fast Data Transformation in R
dependent packages 19total releases 38latest release December 07, 2023most recent commit 3 months ago
Etl ⭐ 327
PHP - ETL (Extract Transform Load) data processing library
dependent packages 15total releases 16latest release November 30, 2023most recent commit 3 months agopackagist flow-php/etl} Downloads
Doctrine Dbal Bulk ⭐ 7
Doctrine DBAL Bulk Operations for selected database engines
dependent packages 1total releases 16latest release November 30, 2023most recent commit 3 months agopackagist flow-php/doctrine-dbal-bulk} Downloads
Pipe21 ⭐ 10
Simple functional pipes
dependent packages 4total releases 45latest release November 30, 2023most recent commit 4 months agopypi pipe21} Downloads
Satpy ⭐ 980
Python package for earth-observing satellite data processing
dependent packages 14total releases 91latest release November 29, 2023most recent commit 3 months agopypi satpy} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.