Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for spark airflow
airflow
x
spark
x
53 search results found
Pipeline
⭐
4,158
PipelineAI
Dataspherestudio
⭐
2,860
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Around Dataengineering
⭐
926
A Data Engineering & Machine Learning Knowledge Hub
Goodreads_etl_pipeline
⭐
593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Data Engineering Interview Questions
⭐
554
More than 2000+ Data engineer interview questions.
Agile_data_code_2
⭐
435
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Data Engineering Projects
⭐
322
Personal Data Engineering Projects
Compass
⭐
284
Compass is a task diagnosis platform for bigdata
Beginner_de_project
⭐
276
Beginner data engineering project - batch edition
Airflow Pipeline
⭐
168
An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR
Movalytics Data Warehouse
⭐
116
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Streamify
⭐
97
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
Data Engineering Nanodegree
⭐
76
Projects done in the Data Engineering Nanodegree by Udacity.com
Airflow Spark Operator Plugin
⭐
66
A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator
Airflow Spark
⭐
64
Docker with Airflow and Spark standalone cluster
Cloud Bigdata Book
⭐
53
write book
Works With Determined
⭐
43
This repository contains example integrations between Determined and other ML products
Udacity Data Engineering
⭐
42
Udacity Data Engineering Nano Degree (DEND)
Pyjaws
⭐
36
PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows
Debussy_concert
⭐
29
Debussy is an opinionated Data Architecture and Engineering framework, enabling data analysts and engineers to build better platforms and pipelines.
Data Engineer Nanodegree Projects Udacity
⭐
27
Projects done in the Data Engineer Nanodegree Program by Udacity.com
Data Engineering Nanodegree
⭐
27
Solution to all projects of Udacity's Data Engineering Nanodegree: Data Modeling with Postgres & Cassandra, Data Warehouse with Redshift, Data Lake with Spark and Data Pipeline with Airflow.
Forklift
⭐
22
🚚 ETL for Spark and Airflow
Jobanalytics_and_search
⭐
22
JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.
Insight Gdelt Feed
⭐
19
A way for home buyers to know about factors affecting a state
Airflow Livy Operators
⭐
17
Lets Airflow DAGs run Spark jobs via Livy: sessions and/or batches.
T Watch
⭐
15
Real Time Twitter Sentiment Analysis Product
Telemetry Streaming
⭐
15
Spark Streaming ETL jobs for Mozilla Telemetry
Ghcn D
⭐
14
Data Pipeline from the Global Historical Climatology Network DataSet
Bigkube
⭐
14
Minikube for big data with Scala and Spark
Bootcamp_data Engineering
⭐
13
Bootcamp to learn basics in Data Engineering
Airflowjob
⭐
11
Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE
Git Influencer
⭐
11
Insight Data Engineering project: A platform built in HDFS, Spark and Airflow to help you to find social influencers from GitHub Network.
Labtools K8s
⭐
10
Complete data engineering pipeline running on Minikube Kubernetes, Spark, Trino, S3, Delta lake, Postgres+ Debezium CDC, Airflow, Kafka Strimzi, Datahub, Zeppelin, Jupyter
Cassandra.lunch
⭐
9
Resources from weekly Zoom lunches revolving around Apache Cassandra and Apache Cassandra-related topics. Hosted by Anant Corporation.
Airflow
⭐
8
This set of code and instructions has the porpouse to instanciate a compiled environment with set of docker images like airflow webserver, airflow scheduler, postgresql, pyspark, Data Pipeline consuming data from weather api , processing with pyspark and storing in postgresql
Docker Bigdata Cluster
⭐
7
Infrastructure automation to deploy Hadoop,Hive,Spark,airflow nodes on a docker host
Aws Etl
⭐
7
This is an ETL application on AWS with general open sales and customer data that you can find here: https://github.com/camposvinicius/data/blob/main/A it's a zipped file with some .csvs inside that we will apply transformations.
Meetup Spark Airflow Demo
⭐
7
Spark & Airflow demo for meetup
Tfl Bikes Data Pipeline
⭐
6
Processing TFL data for bike usage with Google Cloud Platform.
Big Data Cluster
⭐
6
The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center and pgAdmin. This cluster is solely intended for usage in a development environment. Do not use it to run any production workloads.
Spark Mesos Airflow Tutorial
⭐
6
Data Engineer Portfolio
⭐
6
This is a repository to demonstrate my details, skills, projects and to keep track of my progression in Data Analytics and Data Science topics.
Spotless_recommender
⭐
5
Lakehouse
⭐
5
Playground for Lakehouse (Iceberg, Hudi, Spark, Flink, Trino, DBT, Airflow, Kafka, Debezium CDC)
Mapr Airflow
⭐
5
Aws Oss Alternatives
⭐
5
Open Source Alternatives to AWS Services
Udacity Data Engineering Projects
⭐
5
My solutions for the Udacity Data Engineering Nanodegree
Real Estate Sale Analytics
⭐
5
Create data pipeline using Lambda architecture with Spark, Kafka, Airflow and Snowflake
Taxioptimizer
⭐
5
My Data Engineering project @ Insight Data Science
Udacity Data Engineering Nanodegree
⭐
5
This is a repository to hold the files and notebooks produced throughout my Udacity's Nanodegree Data Engineering program.
Airflow Dags
⭐
5
Steam Data Engineering
⭐
5
A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!
Related Searches
Scala Spark (3,279)
Python Spark (2,053)
Java Spark (1,587)
Apache Spark (1,207)
Spark Hadoop (1,188)
Jupyter Notebook Spark (1,151)
Spark Kafka (985)
Spark Streaming (817)
Spark Pyspark (812)
Shell Spark (709)
1-53 of 53 search results
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.