Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Data Lake Open Source Projects

Open source projects categorized as Python Data Lake
Categories > Data Lake
Deeplake ⭐ 7,663
Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai
dependent packages 31total releases 118most recent commit 11 days agopypi deeplake} Downloads
Lakesoul ⭐ 2,248
LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.
dependent packages 1total releases 7most recent commit 3 months ago
Dlt ⭐ 1,069
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
dependent packages 3total releases 52most recent commit 3 months agopypi python-dlt} Downloads
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Aws Serverless Data Lake Framework ⭐ 379
Enterprise-grade, production-hardened, serverless data lake on AWS
most recent commit 3 months ago
Amazon S3 Find And Forget ⭐ 223
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
most recent commit 3 months ago
Hivemq Mqtt Tensorflow Kafka Realtime Iot Machine Learning Training Inference ⭐ 159
Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required
most recent commit 3 years ago
Uncoder_io ⭐ 81
An IDE and translation engine for detection engineers and threat hunters. Be faster, write smarter, keep 100% privacy.
most recent commit 3 months ago
Apachespark ⭐ 59
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
most recent commit 2 years ago
Dataligo ⭐ 47
A library to accelerate ML and ETL pipeline by connecting all data sources
total releases 3most recent commit a year ago
Datapipelines Essentials Python ⭐ 45
Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
most recent commit a year ago
Pan Cortex Data Lake Python ⭐ 32
Python idiomatic SDK for Cortex™ Data Lake.
total releases 15most recent commit 2 years ago
Aws Auto Terminate Idle Emr ⭐ 26
AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.
most recent commit 3 years ago
Tickit Data Lake Demo ⭐ 23
Resources for video demonstrations and blog posts related to DataOps on AWS
most recent commit 2 years ago
Nodestream ⭐ 23
A Fast, Declarative, and Extensible ETL Framework for Graph Databases.
most recent commit 3 months ago
Jobanalytics_and_search ⭐ 22
JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.
most recent commit 2 years ago
Analyzing Reddit Sentiment With Aws ⭐ 16
Learn how to use Kinesis Firehose, AWS Glue, S3, and Amazon Athena by streaming and analyzing reddit comments in realtime. 100-200 level tutorial.
most recent commit 4 years ago
Data Engineering Mta Turnstile ⭐ 14
Data Engineering - Metropolitan Transportation Authority (MTA) Subway Data Analysis
most recent commit 3 months ago
Lakefs Hooks ⭐ 10
a simple lakeFS webhook for pre-commit and pre-merge validation of data objects
most recent commit 7 months ago
Aws Insurancelake Etl ⭐ 7
This solution helps you deploy ETL processes and data storage resources to create an Insurance Lake using Amazon S3 buckets for storage, AWS Glue for data transformation, and AWS CDK Pipelines. It is originally based on the AWS blog Deploy data lake ETL jobs using CDK Pipelines, and complements the InsuranceLake Infrastructure project
most recent commit 3 months ago
Formacao Engenheiro De Dados Cloud E Big Data Azure Databricks ⭐ 6
Formação Engenheiro de Dados Cloud e Big Data (Azure & DataBricks)
most recent commit 6 months ago
Spark Streaming In Python ⭐ 5
Apache Spark 3 - Structured Streaming Course Material
most recent commit 4 years ago
Microsoft Big Data Scientist And Ai ⭐ 5
Microsoft Big Data, Data Scientist, and AI
most recent commit 3 years ago
The 3 Most Depended On Python Data Lake Open Source Projects
Dlt ⭐ 1,069
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
dependent packages 3total releases 52latest release April 18, 2023most recent commit 3 months agopypi python-dlt} Downloads
The 5 Latest Releases In Python Data Lake Open Source Projects
Dlt ⭐ 1,069
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
dependent packages 3total releases 52latest release April 18, 2023most recent commit 3 months agopypi python-dlt} Downloads
Dataligo ⭐ 47
A library to accelerate ML and ETL pipeline by connecting all data sources
total releases 3latest release April 07, 2023most recent commit a year ago
Pan Cortex Data Lake Python ⭐ 32
Python idiomatic SDK for Cortex™ Data Lake.
total releases 15latest release January 09, 2020most recent commit 2 years ago
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.