Awesome Open Source

Programming Languages

Search results for pipeline data flow

8 search results found

Apache Beam is a unified programming model for Batch and Streaming data processing.

Professional Services ⭐ 2,635

Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product.

Nextflow ⭐ 2,447

A DSL for data-driven computational pipelines

Datakit ⭐ 1,044

Connect processes into powerful data pipelines with a simple git-like filesystem interface

Scipipe ⭐ 1,010

Robust, flexible and resource-efficient pipelines using Go and the commandline

Dataflowjavasdk ⭐ 853

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

Purefunctionpipelinedataflow ⭐ 436

My Blog: The Math-based Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with principle-based Warehouse/Workshop Model

Blazing fast NIO microframework and Http Parser

The easiest way to use Machine Learning. Mix and match underlying ML libraries and data set sources. Generate new datasets or modify existing ones with ease.

Dspatch ⭐ 194

The Refreshingly Simple Cross-Platform C++ Dataflow / Patching / Pipelining / Graph Processing / Stream Processing / Reactive Programming Framework

Dataflow ⭐ 169

Efficient Data Loading Pipeline in Pure Python

Dataflowpythonsdk ⭐ 157

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

Dataflowsdk Examples ⭐ 148

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines. This repository hosts a few example pipelines to get you started with Dataflow.

Greenpipes ⭐ 143

An asynchronous pipe implementation for the Task Parallel Library

Awesome Beam ⭐ 130

A curated list of awesome resources for Apache Beam

Mlr3pipelines ⭐ 127

Dataflow Programming for Machine Learning in R

Eventkit ⭐ 119

Event-driven data pipelines

Gcp Variant Transforms ⭐ 114

GCP Variant Transforms

Intelmq Manager ⭐ 101

IntelMQ Manager is a graphical interface to manage configurations for IntelMQ framework.

Dataflow Ops ⭐ 97

Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate

Clojure ++ Apache Beam ++ Google Cloud Dataflow

Dlp Dataflow Deidentification ⭐ 80

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP

Automaticanalysis ⭐ 70

Automatic Analysis (aa)

Pubsub To Bigquery ⭐ 64

A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub

Bigquery Etl Dataflow Sample ⭐ 62

Dataflowtemplate ⭐ 59

Mercari Dataflow Template

Processing Logs Using Dataflow ⭐ 56

Processing Logs at Scale using Cloud Dataflow

Dspatcher ⭐ 55

Cross-Platform Graphical Tool for DSPatch

Prefect Deployment Patterns ⭐ 48

Code examples showing flow deployment to various types of infrastructure

Gameanalytics ⭐ 39

A fully-managed pipeline for game analytics

Plumbermanager ⭐ 39

A helper tool to design CG Pipeline interactive diagrams and data flow documentation

Gae Dataflow ⭐ 38

An example that shows how to periodically launch a Dataflow analytics pipeline from GAE Flex, that reads from Datastore.

Opinionated serverless event analytics pipeline

Auto Data Tokenize ⭐ 36

Identify and tokenize sensitive data automatically using Cloud DLP and Dataflow

Dataflow Java ⭐ 35

Google Cloud Dataflow pipelines such as Identity-By-State as well as useful utility classes.

Healthcare Data Harmonization Dataflow ⭐ 34

Dataflow Video Analytics ⭐ 30

Video Clip Analytics By Using Dataflow and Video AI For Object Tracking

Spark Examples ⭐ 26

Spark pipelines that correspond to a series of Dataflow examples.

Nanostream Dataflow ⭐ 25

real-time stream processing of DNA nanopore sequencer reads with dataflow

Foxsec Pipeline ⭐ 24

Log analysis pipeline utilizing Apache Beam

Dspatchables ⭐ 22

DSPatch Component Repository

Event Streams Dataflow ⭐ 22

Swint Pipe ⭐ 22

Pipeline logic flow for Swint

Python Beam Dataflow Cron ⭐ 19

Base project for creating Python Apache Beam pipelines and running them in Google DataFlow using CRON scheduler

yet another databus that transfer/transform pipeline data between plugins

Gcp Batch Ingestion Bigquery ⭐ 17

Uses Cloud Build to deploy a scalable batch ingestion pipeline consisting of GCS, Cloud Functions, Dataflow and BigQuery

Nuts Flow ⭐ 17

A simple dataflow framework in Python

Zounds is a dataflow library for building directed acyclic graphs that transform audio. It uses the featureflow library to define the processing pipelines.

Go Bitflow ⭐ 16

Bitflow's dataflow engine, a lightweight framework for real-time processing of data streams through a graph of operators

Metaworks ⭐ 16

MetaWorks is a flexible multi-marker metabarcode pipeline for processing paired-end Illumina reads from raw fastq.gz files to taxonomic assignments.

Richflow ⭐ 13

A Node.js and JavaScript synchronous data pipeline processing, data sharing and stream processing library. Actionable & Transformable Pipeline data processing.

Compile your pipeline once, run anywhere

coroutines for dataflow pipelines

Stackexchange Spark Scala Analyser ⭐ 10

Bigquery Ingest Avro Dataflow Sample ⭐ 10

Stream Avro SpecificRecord objects in BigQuery using Cloud Dataflow

Unified programming model to create a data processing pipelines for batch and streaming models

Dataflowpipelinebuilder ⭐ 9

Build TPL Dataflow pipelines using a fluent API and helpful extensions

Dlp Rdb Bq Import ⭐ 8

Relational Database Import to Big Query with Dataflow and DLP API

Generic Event Parser ⭐ 8

This project is a Google Dataflow pipeline that process generic JSON messages from Google PubSub or Apache Kafka and writes it parsed to Google BigQuery.

Pipe Template Python ⭐ 7

Template for dataflow pipelines using python

Apache Beam Example ⭐ 7

Apache Beam Example 中国开源社区

Dataflow Precipitation Pipeline ⭐ 7

Spark Streaming Twitter ⭐ 7

Building pipeline to process the real-time data using Spark and Mongodb.

Sql To Bigquery Dataflow ⭐ 6

This project contains a basic pipeline for migrating a MS SQL Server catalog to a BigQuery dataset.

Datapipeline ⭐ 6

.NET pipeline/dataflow data processing framework written in C#

Dataflowsme ⭐ 6

Tutorial for Cloud Dataflow

Dataflow_template ⭐ 5

Python dataflow pipeline

Dai Deployment Examples ⭐ 5

Examples for deploying Driverless AI (DAI) scorers.

Dsdevent1 ⭐ 5

Data Science Delhi

Dataflowsme Python ⭐ 5

Dataflow Scala Streaming ⭐ 5

Example with Scala and Dataflow for the Madrid Scala Meetup

Real-Time meets High-Delay: A hybrid scheduling and dataflow framework for DSP applications

Related Searches

Python Pipeline (4,255)

Javascript Pipeline (1,369)

Pipeline Jenkins (1,150)

Shell Pipeline (1,143)

Docker Pipeline (1,018)

Jupyter Notebook Pipeline (976)

Java Pipeline (868)

Golang Pipeline (629)

1-8 of 8 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.