Awesome Open Source

Programming Languages

The Top 23 Data Processing Open Source Projects

Open source projects categorized as Data Processing

Categories > Data Processing > Data Processing

Edit Category

Miller ⭐ 8,397

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

total releases 65most recent commit a year ago

Dasel ⭐ 7,495

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

dependent packages 14total releases 97most recent commit 3 months ago

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

dependent packages 4total releases 19most recent commit a month ago

pypi nvidia-dali-cuda110} Downloads

Dialogpt ⭐ 2,365

Large-scale pretraining for dialogue

most recent commit 3 years ago

Texar ⭐ 2,008

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow

total releases 5most recent commit 5 years ago

Data Science On Gcp ⭐ 1,249

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

most recent commit 2 years ago

Bytewax ⭐ 957

Python Stream Processing

dependent packages 3total releases 34most recent commit a year ago

pypi bytewax} Downloads

Numaflow ⭐ 866

Kubernetes-native platform to run massively parallel data/streaming jobs

dependent packages 1total releases 37most recent commit a year ago

Large-scale pretrained models for goal-directed dialog

most recent commit 2 years ago

Dataflowjavasdk ⭐ 853

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

dependent packages 14total releases 38most recent commit 5 years ago

Texar Pytorch ⭐ 711

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

dependent packages 3total releases 5most recent commit 3 years ago

pypi texar-pytorch} Downloads

Hstream ⭐ 668

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

total releases 12most recent commit a year ago

Qualitis ⭐ 631

Qualitis is a one-stop data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing. https://github.com/WeBankFinTech/Qualitis

most recent commit a year ago

Collapse ⭐ 556

Advanced and Fast Data Transformation in R

dependent packages 19total releases 38most recent commit a year ago

Awesome Kafka ⭐ 549

A list about Apache Kafka

most recent commit 2 years ago

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

dependent packages 1total releases 86most recent commit 10 months ago

pypi haupt} Downloads

Eternal ⭐ 433

👾~ music, eternal ~ 👾

most recent commit 3 years ago

Nonechucks ⭐ 315

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

total releases 18most recent commit 4 years ago

Data and tools for generating and inspecting OLMo pre-training data.

total releases 10most recent commit a year ago

Fondant ⭐ 293

Production-ready data processing made easy and shareable

total releases 24most recent commit a year ago

Pysparkling ⭐ 253

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

dependent packages 1total releases 69most recent commit 3 years ago

pypi pysparkling} Downloads

Machine Learning Notebooks ⭐ 241

Machine Learning notebooks for refreshing concepts.

most recent commit 4 years ago

50 Days Of Ml ⭐ 237

A day to day plan for this challenge (50 Days of Machine Learning) . Covers both theoretical and practical aspects

most recent commit 6 years ago

The 10 Most Depended On Data Processing Open Source Projects

Catmandu ⭐ 170

Catmandu - a data processing toolkit

dependent packages 86total releases 150latest release November 06, 2023most recent commit 2 years ago

Collapse ⭐ 556

Advanced and Fast Data Transformation in R

dependent packages 19total releases 38latest release December 07, 2023most recent commit a year ago

Dataflowjavasdk ⭐ 853

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

dependent packages 14total releases 38latest release June 26, 2018most recent commit 5 years ago

Dasel ⭐ 7,495

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

dependent packages 14total releases 97latest release November 28, 2023most recent commit 3 months ago

Forte is a flexible and powerful ML workflow builder. This is part of the CASL project: http://casl-project.ai/

dependent packages 12total releases 14latest release June 29, 2022most recent commit 2 years ago

pypi forte} Downloads

Dataflows ⭐ 182

DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.

dependent packages 11total releases 127latest release October 11, 2023most recent commit 2 years ago

pypi dataflows} Downloads

Biofsharp ⭐ 94

Open source bioinformatics and computational biology toolbox written in F#.

dependent packages 9total releases 18latest release May 11, 2022most recent commit 3 years ago

nuget BioFSharp} Downloads

Datacook ⭐ 38

Machine Learning and Data Analysis in JavaScript.

dependent packages 5total releases 74latest release October 26, 2022most recent commit 3 years ago

npm @pipcook/datacook} Downloads

Barnard59 ⭐ 20

An intuitive and flexible RDF pipeline solution designed to simplify and automate ETL processes for efficient data management.

dependent packages 5total releases 16latest release June 08, 2022most recent commit a year ago

npm barnard59} Downloads

Simple functional pipes

dependent packages 4total releases 45latest release November 30, 2023most recent commit 2 years ago

pypi pipe21} Downloads

The 10 Latest Releases In Data Processing Open Source Projects

Fondant ⭐ 293

Production-ready data processing made easy and shareable

total releases 24latest release December 12, 2023most recent commit a year ago

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

dependent packages 1total releases 86latest release December 10, 2023most recent commit 10 months ago

pypi haupt} Downloads

Collapse ⭐ 556

Advanced and Fast Data Transformation in R

dependent packages 19total releases 38latest release December 07, 2023most recent commit a year ago

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

dependent packages 4total releases 19latest release December 06, 2023most recent commit a month ago

pypi nvidia-dali-cuda110} Downloads

Simple functional pipes

dependent packages 4total releases 45latest release November 30, 2023most recent commit 2 years ago

pypi pipe21} Downloads

Dasel ⭐ 7,495

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

dependent packages 14total releases 97latest release November 28, 2023most recent commit 3 months ago

Pulsar Spark ⭐ 103

Spark Connector to read and write with Pulsar

dependent packages 2total releases 10latest release November 06, 2023most recent commit 2 years ago

Catmandu ⭐ 170

Catmandu - a data processing toolkit

dependent packages 86total releases 150latest release November 06, 2023most recent commit 2 years ago

Artifician ⭐ 8

Artifician is an event-driven framework designed to simplify and accelerate the process of preparing datasets for Artificial Intelligence models.

total releases 32latest release November 04, 2023most recent commit a year ago

Numaflow ⭐ 866

Kubernetes-native platform to run massively parallel data/streaming jobs

dependent packages 1total releases 37latest release November 03, 2023most recent commit a year ago

Popular Data Processing Categories

Jupyter Notebook

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2025 Awesome Open Source. All rights reserved.