Awesome Open Source
Awesome Open Source
Combined Topics
parquet x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 146 Parquet Open Source Projects on Github

Categories > Data Processing > Parquet
Gaffer ⭐ 1,635
A large-scale entity and relation database supporting aggregation of properties
Drill ⭐ 1,594
Apache Drill is a distributed MPP query layer for self describing data
Parquet Mr ⭐ 1,384
Apache Parquet
Petastorm ⭐ 1,223
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Quilt ⭐ 1,072
Quilt is a self-organizing data hub for S3
Parquet Format ⭐ 890
Apache Parquet
Devops Python Tools ⭐ 463
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Choetl ⭐ 427
ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Roapi ⭐ 424
Create full-fledged APIs for static datasets without writing a single line of code.
Iceberg ⭐ 409
Iceberg is a table format for large, slow-moving tabular data
Skale ⭐ 392
High performance distributed data processing engine
Oap ⭐ 362
Optimized Analytics Package for Spark* Platform
Pystore ⭐ 331
Fast data store for Pandas time-series data
Elasticsearch_loader ⭐ 319
A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch
Parquet Cpp ⭐ 312
Apache Parquet
Vscode Data Preview ⭐ 298
Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files
Ratatool ⭐ 293
A tool for data sampling, data generation, and data diffing
Parquet Dotnet ⭐ 226
🏐 Apache Parquet for modern .NET
Awkward 0.x ⭐ 215
Manipulate arrays of complex data structures as easily as Numpy.
Parquetviewer ⭐ 202
Simple windows desktop application for viewing & querying Apache Parquet files
Parquetjs ⭐ 189
fully asynchronous, pure JavaScript implementation of the Parquet file format
Kglab ⭐ 162
Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, RDFlib, pySHACL, RAPIDS, NetworkX, iGraph, PyVis, pslpython, pyarrow, etc.
Bigdata Playground ⭐ 154
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Kartothek ⭐ 154
A consistent table management library in python
Parquet4s ⭐ 149
Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.
Sqlite Parquet Vtable ⭐ 146
A SQLite vtable extension to read Parquet files
Parquet Go ⭐ 143
Go package to read and write parquet files. parquet is a file format to store nested data structures in a flat columnar data format. It can be used in the Hadoop ecosystem and with tools such as Presto and AWS Athena.
Eel Sdk ⭐ 140
Big Data Toolkit for the JVM
Amazon S3 Find And Forget ⭐ 129
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
Parquet Rs ⭐ 129
Apache Parquet implementation in Rust
Parquet Index ⭐ 113
Spark SQL index for Parquet tables
Bigdata File Viewer ⭐ 92
A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
Schemer ⭐ 89
Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.
Sparksql Protobuf ⭐ 73
Read SparkSQL parquet file as RDD[Protobuf]
Rumble ⭐ 68
⛈️ RumbleDB 1.15.0 "Ivory Palm" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
Parquet2 ⭐ 64
Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow
Gcs Tools ⭐ 60
GCS support for avro-tools, parquet-tools and protobuf
Parquet.jl ⭐ 57
Julia implementation of parquet columnar file format reader
Spark ⭐ 55
Apache Spark is a fast, in-memory data processing engine with elegant and expressive development API's to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets.This project will have sample programs for Spark in Scala language .
Monix Connect ⭐ 52
A set of connectors for Monix. 🔛
Node Parquet ⭐ 44
NodeJS module to access apache parquet format files
Etl Light ⭐ 38
A light Kafka to HDFS/S3 ETL library based on Apache Spark
Odbc2parquet ⭐ 38
A command line tool to query an ODBC data source and write the result into a parquet file.
Paraflow ⭐ 36
A real-time analytical system for ID-associated data
Pqrs ⭐ 33
Command line tool for inspecting Parquet files
Csv2parquet ⭐ 33
Convert a CSV to a parquet file.
Miniparquet ⭐ 33
Library to read a subset of Parquet files
Parquet Extra ⭐ 31
A collection of Apache Parquet add-on modules
Parquet ⭐ 29
A library for reading and writing parquet files.
Pucket ⭐ 29
Bucketing and partitioning system for Parquet
Meepo ⭐ 27
异构存储数据迁移
Parquet Flinktacular ⭐ 26
How to use Parquet in Flink
Daflow ⭐ 24
Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
Intellij Avro Parquet Plugin ⭐ 23
A Tool Window plugin for IntelliJ that displays Avro and Parquet files and their schemas in JSON.
Columnify ⭐ 22
Make record oriented data to columnar format.
Terraform Aws Kinesis Firehose ⭐ 21
This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.
Obslytics ⭐ 21
Tools and Services allowing seamless usage of Observability data from Prometheus, Thanos, Cortex, M3DB, Loki and more!
Parquet Tools ⭐ 20
easy install parquet-tools
Imctermite ⭐ 18
Enables extraction of measurement data from binary files with extension 'raw' used by proprietary software imcFAMOS/imcSTUDIO and facilitates its storage in open source file formats
Graphique ⭐ 17
GraphQL service for arrow tables and parquet data sets.
Albis ⭐ 17
Albis: High-Performance File Format for Big Data Systems
Sparrow ⭐ 16
Temporary Shorcut For Reading Arrow/Parquet Bits Into R via 'reticulate'
Experiments ⭐ 15
Code examples for my blog posts
Parquet Generator ⭐ 15
Parquet file generator
Wasp ⭐ 14
WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.
Dumping Machine ⭐ 12
Dumping Machine is an application which dumps Kafka Avro topics to S3 or HDFS as Parquet
Parquet Usql ⭐ 12
A custom extractor designed to read parquet for Azure Data Lake Analytics
Databricks Notebooks ⭐ 11
Collection of Databricks and Jupyter Notebooks
Embulk Output S3_parquet ⭐ 11
Embulk (https://github.com/embulk/embulk/) output plugin to dump records as Apache Parquet (https://parquet.apache.org/) files on S3.
Tableio.jl ⭐ 11
A glue package for reading and writing tabular data. It aims to provide a uniform api for reading and writing tabular data from and to multiple sources.
Huemul Bigdatagovernance ⭐ 10
Huemul BigDataGovernance, es una framework que trabaja sobre Spark, Hive y HDFS. Permite la implementación de una estrategia corporativa de dato único, basada en buenas prácticas de Gobierno de Datos. Permite implementar tablas con control de Primary Key y Foreing Key al insertar y actualizar datos utilizando la librería, Validación de nulos, largos de textos, máximos/mínimos de números y fechas, valores únicos y valores por default. También permite clasificar los campos en aplicabilidad de derechos ARCO para facilitar la implementación de leyes de protección de datos tipo GDPR, identificar los niveles de seguridad y si se está aplicando algún tipo de encriptación. Adicionalmente permite agregar reglas de validación más complejas sobre la misma tabla.
Avro Cli ⭐ 9
Yet Another Avro CLI Tool
Stata Parquet ⭐ 9
Read and write parquet files from Stata
Parquet Testing ⭐ 9
Auxiliary files for compatibility and integration tests for Apache Parquet
Drill Test Framework ⭐ 9
Test Framework for Apache Drill
Parquet Metadata ⭐ 9
Dump metadata about a Parquet file.
Pdf2dataset ⭐ 8
Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features
Cda Client ⭐ 8
Cloud Data Access client
Hdfsreader ⭐ 8
parquet for DataX - hdfsreader
Php Parquet ⭐ 7
PHP implementation for reading and writing Apache Parquet files/streams
Spark Streaming Twitter ⭐ 7
Building pipeline to process the real-time data using Spark and Mongodb.
Score ⭐ 7
ScORe - Programmatic Schema On Read for Spark SQL, powered by Taboola
Pyspark Dataframe Made Easy ⭐ 7
pyspark dataframe made easy
Queryeverythingwithsql ⭐ 6
Query every thing by sqlite virtual table
Xpq ⭐ 6
Simple command line tool for analyzing parquet files
Ob_datastash ⭐ 6
Stream your CSV files to an HTTP API
Cloud Storage Extension ⭐ 5
Exasol Cloud Storage Extension for accessing formatted data Avro, Orc and Parquet, on public cloud storage systems
Streamsx.parquet ⭐ 5
(Incubation) Toolkit providing adapters to Parquet
Kafka Replicator ⭐ 5
Kafka replicator is a tool used to mirror and backup Kafka topics across regions
Nifi Cdsw Edge ⭐ 4
Edge to AI: IoT
Wspr Analytics ⭐ 4
Big Data utilities for running queries on WSPR DataSets using Apache Arrow, Spark, PySpark, Scala and Java on CSV, Parquet or Avro file formats.
Csv2parquet2orc ⭐ 4
CSV 2 Parquet and CSV2 to ORC converter with aligned interface
Embulk Input Parquet_hadoop ⭐ 4
Parquet input plugin for Embulk
Aporia Importer ⭐ 4
🏋️‍♀️ Import inference data from Amazon S3, Azure Blob Storage, Google Cloud Storage and others to Aporia
Hadoop Small Files Merger ⭐ 4
A Spark application to merge small files on Hadoop
Datatools ⭐ 4
DataTools in Python
Csv_to_parquet_converter ⭐ 4
csv to parquet and vice versa file converter based on Pandas written in Python3
Docker Hadoop Hive Parquet ⭐ 3
Hadoop, Hive, Parquet and Hue in docker-compose v3
Splumbr ⭐ 3
Sql Benchmarks ⭐ 3
A set of Spark SQL benchmarks
1-100 of 146 projects
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021