Awesome Open Source

Programming Languages

Search results for pipeline apache

35 search results found

Apache Beam is a unified programming model for Batch and Streaming data processing.

Dataflowjavasdk ⭐ 853

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

Hop Orchestration Platform

Data Pipelines With Apache Airflow ⭐ 587

Code for Data Pipelines with Apache Airflow

piper - a distributed workflow engine

Sparkflow ⭐ 301

Easy to use library to bring Tensorflow on Apache Spark

Jpmml Sparkml ⭐ 265

Java library and command-line application for converting Apache Spark ML pipelines to PMML

Fast iterative local development and testing of Apache Airflow workflows

Airflow_for_beginners ⭐ 166

Dataflowpythonsdk ⭐ 157

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

Dataflowsdk Examples ⭐ 148

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines. This repository hosts a few example pipelines to get you started with Dataflow.

Envelope ⭐ 133

Build configuration-driven ETL pipelines on Apache Spark

Incubator Liminal ⭐ 131

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

Awesome Beam ⭐ 130

A curated list of awesome resources for Apache Beam

Mirror of Apache Crunch (Incubating)

Mirror of Apache Falcon

Flink Connectors ⭐ 93

Apache Flink connectors for Pravega.

Pyspark2pmml ⭐ 93

Python library for converting Apache Spark ML pipelines to PMML

Aws Concurrent Data Orchestration Pipeline Emr Livy ⭐ 66

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

Data Stream Development With Apache Spark Kafka And Spring Boot ⭐ 54

Data Stream Development with Apache Spark, Kafka and Spring Boot by Packt Publishing

Lighthouse ⭐ 54

Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

Deep Learning Pyspark ⭐ 40

Deep Learning with Apache Spark and Deep Cognition

Ml In Production ⭐ 39

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

Beam Learning Month ⭐ 34

Spark Flow ⭐ 32

Library for organizing batch processing pipelines in Apache Spark

Deploy a secured, clustered, auto-scaling NiFi service in AWS.

Cogstack Nifi ⭐ 29

Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

Odsc_india_2018 ⭐ 26

My presentation at ODSC India 2018 about Deep Learning with Apache Spark

Ctakes Clinical Pipeline ⭐ 21

Clinical Pipeline Engine using Apache cTAKES

Pipeline ⭐ 21

Free, open-source software for crowdsourcing creative projects

Python Beam Dataflow Cron ⭐ 19

Base project for creating Python Apache Beam pipelines and running them in Google DataFlow using CRON scheduler

Twitter Stream ⭐ 18

Twitter-Kafka Data Pipeline

Sparklanes ⭐ 16

A lightweight data processing framework for Apache Spark

Apachekafka ⭐ 16

A curated re-sources list for awesome Apache Kafka

Airflow Ci ⭐ 15

Apache Airflow CI pipeline

Data Factory R Server Apache Spark Pipeline ⭐ 15

This tutorial highlights how to build a scalable machine-learning based data processing pipeline using Microsoft R Server with Apache Spark utilizing Azure Data Factory (ADF)

Graphsense Transformation ⭐ 15

GraphSense Transformation Pipeline

Virapipe ⭐ 13

ViraPipe is a Apache Spark based scalable pipeline for metagenome analysis from NGS read data

Deepvariant On Spark ⭐ 11

DeepVariant-on-Spark is a germline short variant calling pipeline that runs Google DeepVariant on Apache Spark at scale.

Stackexchange Spark Scala Analyser ⭐ 10

Unified programming model to create a data processing pipelines for batch and streaming models

Spark Pipeline ⭐ 9

Machine learning pipeline for Apache Spark

Beam Scala Examples ⭐ 9

Scala examples for using Apache Beam Java API (2.1.0)

Spark Pipeline ⭐ 9

Example End-to-End Data Pipeline with Apache Spark from Data Analysis to Data Product

Apache Spark Etl Pipeline Example ⭐ 8

Demonstration of using Apache Spark to build robust ETL pipelines while taking advantage of open source, general purpose cluster computing.

Additional functions useful for building Apache Beam pipelines in Go.

Pipeline Docs ⭐ 8

Some Pipeline.IO Usecase Documentation

Kafka Twitter Fanout ⭐ 7

Part of a data processing pipeline example using Apache Kafka on Heroku

Apache Beam Example ⭐ 7

Apache Beam Example 中国开源社区

Docker Streamsets Kafka Minecraft ⭐ 7

Visualize Apache logs in Minecraft using Docker, Streamsets Data Collector, Spigot and Kafka .

Real Time Stock Streaming Pipeline ⭐ 6

A set of helpers to build Apache Spark pipelines for Neuroimaging

Streamingpipeline ⭐ 6

Stream processing pipeline with apache beam, druid and superset

Sparklyr2pmml ⭐ 6

R library for converting Apache Spark ML pipelines to PMML

Dcat Ap Viewer ⭐ 6

Viewer of DCAT-AP 2.0.1 compatible dataset metadata

Kafka Twitter Aggregate ⭐ 5

Part of a data processing pipeline example using Apache Kafka on Heroku

Beam Errorhandle Example ⭐ 5

Simple example of error handling per-element in Apache Beam.

A simple in-memory, configuration driven, data processing pipeline for Apache Spark.

Machine Learning Pipeline Lr Pyspark ⭐ 5

Power Plant ML Pipeline Application - Apache Spark

Related Searches

Java Apache (4,331)

Python Pipeline (4,255)

Php Apache (2,627)

Javascript Apache (1,555)

Python Apache (1,541)

Shell Apache (1,492)

Javascript Pipeline (1,369)

Docker Apache (1,277)

Apache Spark (1,207)

Pipeline Jenkins (1,150)

1-35 of 35 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.