Awesome Open Source

Programming Languages

Search results for airflow pyspark

12 search results found

Gather Deployment ⭐ 351

Gathers Python deployment, infrastructure and practices.

Movalytics Data Warehouse ⭐ 117

Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

Python_mozetl ⭐ 26

ETL jobs for Firefox Telemetry

Jobanalytics_and_search ⭐ 22

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

This set of code and instructions has the porpouse to instanciate a compiled environment with set of docker images like airflow webserver, airflow scheduler, postgresql, pyspark, Data Pipeline consuming data from weather api , processing with pyspark and storing in postgresql

This is an ETL application on AWS with general open sales and customer data that you can find here: https://github.com/camposvinicius/data/blob/main/A it's a zipped file with some .csvs inside that we will apply transformations.

Reddit Data Engineering ⭐ 7

An end-to-end data engineering pipeline to create a dashboard for the latest content on the r/Stocks subreddit

Airflow Pyspark Emr ⭐ 7

This project demonstrate how to process data stored in a data lake fashion, transforming it into an OLAP optimized structure by using PySpark. The PySpark Job runs on AWS EMR, and the Data Pipeline is orchestrated by Apache Airflow, including the infrastructure creation and the EMR cluster termination.

Spark Mesos Airflow Tutorial ⭐ 6

Big Data Cluster ⭐ 6

The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center and pgAdmin. This cluster is solely intended for usage in a development environment. Do not use it to run any production workloads.

Datasprints Open Spaces ⭐ 5

Repository for the code demoed in the talk

Related Searches

Spark Pyspark (773)

Python Pyspark (689)

Python Airflow (681)

1-12 of 12 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.