Awesome Open Source
Awesome Open Source
Combined Topics
parquet x

The Top 643 Parquet Open Source Projects

Categories > Data Processing > Parquet
Aws Data Wrangler ⭐ 2,819
Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
dependent packages 28total releases 119most recent commit 2 days agopypi awswrangler} Downloads
Iceberg ⭐ 2,793
Apache Iceberg
most recent commit 4 hours ago
Dsq ⭐ 1,956
Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.
most recent commit 21 hours ago
Drill ⭐ 1,676
Apache Drill is a distributed MPP query layer for self describing data
dependent packages 11total releases 20most recent commit 7 hours ago
Gaffer ⭐ 1,655
A large-scale entity and relation database supporting aggregation of properties
dependent packages 10total releases 90most recent commit 6 hours ago
Parquet Mr ⭐ 1,533
Apache Parquet
dependent packages 141total releases 12most recent commit 6 days ago
Petastorm ⭐ 1,413
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
dependent packages 6total releases 77most recent commit a month agopypi petastorm} Downloads
Roapi ⭐ 1,257
Create full-fledged APIs for slowly moving datasets without writing a single line of code.
total releases 17most recent commit 2 months ago
Quilt ⭐ 1,153
Quilt is a self-organizing data hub for S3
dependent packages 20total releases 53most recent commit 3 hours agopypi quilt3} Downloads
Influxdb_iox ⭐ 1,130
Pronounced (influxdb eye-ox), short for iron oxide. This is the new core of InfluxDB written in Rust on top of Apache Arrow.
most recent commit 3 hours ago
Parquet Format ⭐ 1,019
Apache Parquet
dependent packages 12total releases 8most recent commit 2 months ago
Adam ⭐ 925
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
dependent packages 17total releases 14most recent commit 2 days ago
Parquet Go ⭐ 842
pure golang library for reading/writing parquet file
dependent packages 81total releases 40most recent commit 12 days ago
Devops Python Tools ⭐ 550
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
most recent commit 15 days ago
Choetl ⭐ 533
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
dependent packages 7total releases 150most recent commit 15 days agonuget ChoETL.NETStandard} Downloads
Pigpen ⭐ 513
Map-Reduce for Clojure
dependent packages 6total releases 34most recent commit 5 years ago
Tech.ml.dataset ⭐ 439
A Clojure high performance data processing system
total releases 251most recent commit 16 days ago
Sparser ⭐ 411
Sparser: Raw Filtering for Faster Analytics over Raw Data
most recent commit 4 years ago
Iceberg ⭐ 409
Iceberg is a table format for large, slow-moving tabular data
most recent commit 9 months ago
Pystore ⭐ 404
Fast data store for Pandas time-series data
dependent packages 1total releases 35most recent commit 3 months agopypi PyStore} Downloads
Kglab ⭐ 397
Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.
total releases 22most recent commit a month ago
Skale ⭐ 392
High performance distributed data processing engine
dependent packages 1total releases 29most recent commit a year agonpm skale-engine} Downloads
Amadeus ⭐ 334
Harmonious distributed data analysis in Rust.
dependent packages 7total releases 25most recent commit a year agocargo amadeus-core} Downloads
Spindle ⭐ 333
Next-generation web analytics processing with Scala, Spark, and Parquet.
most recent commit 7 years ago
Elasticsearch_loader ⭐ 323
A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch
dependent packages 2total releases 34most recent commit 7 months agopypi elasticsearch-loader} Downloads
Ratatool ⭐ 320
A tool for data sampling, data generation, and data diffing
dependent packages 6total releases 29most recent commit 19 hours ago
Parquet Dotnet ⭐ 319
🏐 Apache Parquet for modern .NET
total releases 4most recent commit 3 months ago
Centurion ⭐ 318
Kotlin Bigdata Toolkit
dependent packages 3total releases 1most recent commit 5 months ago
Parquet Cpp ⭐ 312
Apache Parquet
most recent commit 3 years ago
Vscode Data Preview ⭐ 298
Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files
most recent commit 10 months ago
Parquetviewer ⭐ 284
Simple windows desktop application for viewing & querying Apache Parquet files
most recent commit 3 months ago
Parquet Python ⭐ 263
python implementation of the parquet columnar file format.
dependent packages 4total releases 5most recent commit a year agopypi parquet} Downloads
Rill Developer ⭐ 225
a tool that makes it effortless to transform and explore your datasets with SQL, powered by DuckDB
most recent commit 7 hours ago
Awkward 0.x ⭐ 215
Manipulate arrays of complex data structures as easily as Numpy.
dependent packages 8total releases 6most recent commit a year agopypi awkward0} Downloads
Bigdata File Viewer ⭐ 208
A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
most recent commit 4 days ago
Parquet Go ⭐ 201
Go package to read and write parquet files. parquet is a file format to store nested data structures in a flat columnar data format. It can be used in the Hadoop ecosystem and with tools such as Presto and AWS Athena.
dependent packages 11total releases 43most recent commit a month ago
Parquet4s ⭐ 196
Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.
dependent packages 2total releases 44most recent commit 21 days ago
Parquetjs ⭐ 189
fully asynchronous, pure JavaScript implementation of the Parquet file format
dependent packages 9total releases 7most recent commit a year agonpm parquetjs} Downloads
Spark Programming Guide Zh Cn ⭐ 179
Spark 编程指南简体中文版
most recent commit 6 years ago
Parquet Dotnet ⭐ 176
🐬 Apache Parquet for modern .Net
dependent packages 15total releases 209most recent commit 3 months agonuget Parquet.Net} Downloads
Amazon S3 Find And Forget ⭐ 169
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
most recent commit a month ago
D6tstack ⭐ 166
Quickly ingest messy CSV and XLS files. Export to clean pandas, SQL, parquet
dependent packages 1total releases 11most recent commit 4 months agopypi d6tstack} Downloads
Rumble ⭐ 165
⛈️ RumbleDB 1.18.0 "Scarlet Ixora" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
total releases 4most recent commit a month ago
Tutorials_cn ⭐ 161
most recent commit 6 hours ago
Kartothek ⭐ 160
A consistent table management library in python
total releases 38most recent commit 3 months ago
Bigdata Playground ⭐ 154
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
most recent commit 3 years ago
Impala Tpcds Kit ⭐ 150
TPC-DS Kit for Impala
most recent commit 4 months ago
Parquet2 ⭐ 148
Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow
dependent packages 2total releases 21most recent commit 6 days agocargo parquet2} Downloads
Sqlite Parquet Vtable ⭐ 146
A SQLite vtable extension to read Parquet files
most recent commit 2 years ago
Nyc Transport ⭐ 144
A Unified Database of NYC transport (subway, taxi/Uber, and citibike) data.
most recent commit 5 years ago
Eel Sdk ⭐ 140
Big Data Toolkit for the JVM
dependent packages 17total releases 103most recent commit 2 years ago
Herringbone ⭐ 135
Tools for working with parquet, impala, and hive
most recent commit a year ago
Parquet Rs ⭐ 129
Apache Parquet implementation in Rust
most recent commit 3 years ago
Sergeant ⭐ 124
💂 Tools to Transform and Query Data with 'Apache' 'Drill'
most recent commit 6 months ago
Parquet_fdw ⭐ 119
Parquet foreign data wrapper for PostgreSQL
most recent commit 3 months ago
Parquet Go ⭐ 114
Go library to read/write Parquet files
most recent commit 14 hours ago
Parquet Index ⭐ 113
Spark SQL index for Parquet tables
most recent commit a year ago
Geo Arrow Spec ⭐ 109
Specifications for storing geospatial data in Apache Arrow and Apache Parquet
most recent commit 6 days ago
Spectrify ⭐ 103
Export Redshift data and convert to Parquet for use with Redshift Spectrum or other data warehouses.
total releases 11most recent commit 2 years ago
Streamx ⭐ 95
kafka-connect-s3 : Ingest data from Kafka to Object Stores(s3)
most recent commit 3 years ago
Gpu Bdb ⭐ 93
RAPIDS GPU-BDB
most recent commit 9 days ago
Thermorawfileparser ⭐ 92
Thermo RAW file parser that runs on Linux/Mac and all other platforms that support Mono
most recent commit 9 days ago
Datasets.jl ⭐ 90
most recent commit 5 days ago
Schemer ⭐ 89
Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.
total releases 15most recent commit 2 years ago
Scientificsummarizationdatasets ⭐ 88
Datasets I have created for scientific summarization, and a trained BertSum model
most recent commit 3 years ago
Tensorqtl ⭐ 84
Ultrafast GPU-based QTL mapper
most recent commit 8 days ago
Json2parquet ⭐ 80
Convert JSON files to Parquet using PyArrow
total releases 30most recent commit 2 months ago
Pqrs ⭐ 78
Command line tool for inspecting Parquet files
total releases 3most recent commit 9 days ago
Odbc2parquet ⭐ 75
A command line tool to query an ODBC data source and write the result into a parquet file.
total releases 52most recent commit 8 days ago
Parquetsharp ⭐ 75
ParquetSharp is a .NET library for reading and writing Apache Parquet files.
dependent packages 1total releases 53most recent commit 5 hours agonuget ParquetSharp} Downloads
Oamap ⭐ 74
Perform high-speed calculations on columnar data without creating intermediate objects.
total releases 34most recent commit 4 years ago
Sparksql Protobuf ⭐ 73
Read SparkSQL parquet file as RDD[Protobuf]
most recent commit 4 years ago
Boxball ⭐ 73
Prebuilt Docker images with Retrosheet's complete baseball history data for many analytical frameworks. Includes Postgres, cstore_fdw, MySQL, SQLite, Clickhouse, Drill, Parquet, and CSV.
total releases 3most recent commit a month ago
Hybridbackend ⭐ 71
A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster
total releases 14most recent commit 7 days ago
Parquet.jl ⭐ 68
Julia implementation of parquet columnar file format reader
most recent commit 7 months ago
Dendrite ⭐ 67
Dendrite is a library for querying large datasets on a single host at near-interactive speeds.
total releases 27most recent commit a year ago
Guery ⭐ 61
Distributed SQL query engine written in Go for big data
most recent commit 4 years ago
Rainbow ⭐ 61
A data layout optimization framework for wide tables stored on HDFS. See rainbow's webpage
most recent commit 4 years ago
Avro Parquet Spark Example ⭐ 61
An example of using Avro and Parquet in Spark SQL
most recent commit 7 years ago
Csv2parquet ⭐ 60
Create Parquet files from CSV
most recent commit 5 years ago
Gcs Tools ⭐ 60
GCS support for avro-tools, parquet-tools and protobuf
most recent commit 5 months ago
Parquet Compatibility ⭐ 59
compatibility tests to make sur C and Java implementations can read each other
most recent commit 8 years ago
Parquet Go Source ⭐ 58
source provider for parquet-go
dependent packages 57total releases 10most recent commit 2 months ago
Osm Parquetizer ⭐ 58
A converter for the OSM PBFs to Parquet files
most recent commit 2 years ago
Monix Connect ⭐ 58
A set of connectors for Monix. 🔛
dependent packages 1total releases 11most recent commit 2 days ago
Spark ⭐ 55
Apache Spark is a fast, in-memory data processing engine with elegant and expressive development API's to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets.This project will have sample programs for Spark in Scala language .
most recent commit 4 months ago
Parquet Cli ⭐ 52
Command line (CLI) tool to inspect Apache Parquet files on the go
total releases 4most recent commit 3 years ago
Spark Compaction ⭐ 52
File compaction tool that runs on top of the Spark framework.
most recent commit 3 years ago
Parquet Wasm ⭐ 49
Rust-based WebAssembly bindings to read and write Apache Parquet data
most recent commit 14 days ago
Iceberg ⭐ 47
A temporary home for LinkedIn's changes to Apache Iceberg (incubating)
most recent commit 2 months ago
Spark Mail ⭐ 45
Tutorial on parsing Enron email to Avro and then explore the email set using Spark.
most recent commit 3 years ago
Ibis ⭐ 44
IBIS is a workflow creation-engine that abstracts the Hadoop internals of ingesting RDBMS data.
most recent commit 4 years ago
Spark Parquet Thrift Example ⭐ 44
Example Spark project using Parquet as a columnar store with Thrift objects.
most recent commit 8 years ago
Node Parquet ⭐ 44
NodeJS module to access apache parquet format files
most recent commit 2 years ago
Parquet Go ⭐ 43
Go library to work with PARQUET files
most recent commit 3 years ago
Snowset ⭐ 41
Snowflake dataset containing statistics for 70 million queries over 14 day period
most recent commit 8 months ago
Sfarrow ⭐ 40
R package for reading/writing `sf` objects from/to parquet files with `arrow`.
most recent commit 7 months ago
Ml Io ⭐ 40
A high performance data access library for machine learning tasks
most recent commit 2 years ago
Entrada ⭐ 39
Entrada - A tool for DNS big data analytics
most recent commit a day ago
Etl Light ⭐ 38
A light Kafka to HDFS/S3 ETL library based on Apache Spark
most recent commit 5 years ago
1-100 of 643 projects
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,048,735)
Python  (796,103)
Java  (385,413)
Php  (284,589)
Typescript  (239,722)
C Plus Plus  (236,213)
Ruby  (221,696)
C  (180,396)
Shell  (169,594)
C Sharp  (162,015)
Golang  (156,089)
Swift  (64,698)
R  (57,248)
Objective C  (57,169)
Rust  (56,117)
Dart  (51,466)
Kotlin  (45,511)
Lua  (33,293)
Matlab  (30,891)
Perl  (30,483)
Language  (30,085)
Scala  (28,601)
Haskell  (20,260)
Clojure  (19,072)
Powershell  (18,797)
Bash  (18,606)
Coffeescript  (17,331)
Elixir  (16,565)
Assembly  (14,526)
Processing  (13,298)
Julia  (12,536)
Basic  (10,765)
Flow  (10,528)
Emacs Lisp  (10,368)
Erlang  (9,058)
Scheme  (8,437)
Groovy  (8,431)
Solidity  (7,611)
Ocaml  (6,668)
Pascal  (6,378)
Hack  (6,365)
Lisp  (6,340)
Fortran  (5,483)
Elm  (5,477)
Common Lisp  (5,277)
Dlang  (4,747)
Shell Script  (4,633)
Python Library  (4,446)
Crystal  (4,374)
Scripting  (4,203)
Top Projects
Freecodecamp ⭐ 346,432
996.icu ⭐ 262,320
Free Programming Books ⭐ 233,678
Coding Interview University ⭐ 220,499
Awesome ⭐ 202,578
Vue ⭐ 196,186
Developer Roadmap ⭐ 195,398
Public Apis ⭐ 193,925
React ⭐ 188,417
System Design Primer ⭐ 180,789
Tensorflow ⭐ 165,144
Bootstrap ⭐ 157,483
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,459
Build Your Own X ⭐ 142,246
Flutter ⭐ 140,926
Javascript Algorithms ⭐ 140,055
Python ⭐ 136,303
Gitignore ⭐ 133,450
Linux ⭐ 132,199
Vscode ⭐ 132,056
Awesome Python ⭐ 127,528
Javascript ⭐ 123,347
Javaguide ⭐ 121,667
Python 100 Days ⭐ 117,585
Computer Science ⭐ 113,544
Youtube Dl ⭐ 110,239
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,862
Electron ⭐ 101,813
D3 ⭐ 100,901
Go ⭐ 99,493
30 Seconds Of Code ⭐ 95,528
Create React App ⭐ 95,174
Axios ⭐ 93,562
Awesome Selfhosted ⭐ 89,230
Kubernetes ⭐ 88,689
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,823
Next.js ⭐ 87,114
Terminal ⭐ 83,251
Deno ⭐ 82,597
Three.js ⭐ 82,214
Angular ⭐ 81,516
Awesome Go ⭐ 81,089
Typescript ⭐ 80,876
Ant Design ⭐ 80,416
Material Ui ⭐ 78,638
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.