Awesome Open Source

Programming Languages

Search results for apache beam

45 search results found

TFX is an end-to-end platform for deploying production ML pipelines

Dataflowtemplates ⭐ 1,060

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

Yet Another UserAgent Analyzer

Flink On K8s Operator ⭐ 650

[DEPRECATED] Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

Bitcoin Etl ⭐ 350

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Weather Tools ⭐ 186

Apache Beam pipelines to make weather data accessible and useful.

Flink On K8s Operator ⭐ 177

Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

Tensorflow Recorder ⭐ 158

TFRecorder makes it easy to create TensorFlow records (TFRecords) from Pandas DataFrames and CSVs files containing images or structured data.

Datasplash ⭐ 128

Clojure API for a more dynamic Google Dataflow

Fhir Data Pipes ⭐ 107

A collection of tools for extracting FHIR resources and analytics services on top of that data.

Asgarde allows simplifying error handling with Apache Beam Java, with less code, more concise and expressive code.

Dataflowtemplate ⭐ 59

Mercari Dataflow Template

Mlengine Boilerplate ⭐ 58

Repository to quickly get you started with new Machine Learning projects on Google Cloud Platform. More info(slides):

Data_processing_course ⭐ 53

Some class materials for a data processing course using PySpark

Beam Nuggets ⭐ 52

Collection of transforms for the Apache beam python SDK.

Micro Apps ⭐ 48

Microservices in Post-Kubernetes Era. A polyglot monorepo

Bigquery To Datastore ⭐ 47

Export a whole BigQuery table to Google Datastore with Apache Beam/Google Dataflow

Opinionated serverless event analytics pipeline

Foxsec Pipeline ⭐ 24

Log analysis pipeline utilizing Apache Beam

Pysql Beam ⭐ 23

Beam Mysql Connector ⭐ 20

An io connector of Apache Beam to access MySQL databases.

Dataflowtemplates ⭐ 20

Convenient Dataflow pipelines for transforming data between cloud data sources

Proxima Platform ⭐ 17

The Proxima platform.

Beam Pipeline Examples ⭐ 16

Apache Beam examples for running on Google Cloud Dataflow.

Consent Based Conversion Adjustments ⭐ 15

Code to statistically up-weight conversion values of consenting customers to feed up to 100% of the factual conversion values back into Google Ads.

Kuromoji For Bigquery ⭐ 14

Tokenize Japanese text on BigQuery with Kuromoji in Apache Beam/Google Dataflow at scale

Midgard is a wrapper on Beam Kotlin, allowing more concise and expressive code. It removes Beam boilerplate code and proposes more Functional Programming style

Processing Text Data ⭐ 13

Presents an optimized Apache Beam pipeline for generating sentence embeddings (runnable on Cloud Dataflow).

Dataset_grouper ⭐ 12

Kotlin extensions for Apache Beam

Data Engineering ⭐ 12

Projects and studies regarding Data Engineering Area

Count Tokens Hf Datasets ⭐ 12

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Covid 19 Apache Beam Statistics ⭐ 10

Statistical processing of COVID-19 data using Apache Beam for Google Cloud Dataflow in Python. Project for the exam of "Sistemi ed Applicazioni Cloud" (2019-20), Magistrale di Ingegneria Informatica at the Dipartimento di Ingegneria Enzo Ferrari.

Beam Scala Examples ⭐ 9

Scala examples for using Apache Beam Java API (2.1.0)

Idiomatic Kotlin Pipelines for Apache Beam

Dataflow Bigquery Dynamic Destinations ⭐ 9

An example pipeline for dynamically routing events from Pub/Sub to different BigQuery tables based on a message attribute.

Apache Beam Example ⭐ 9

Apache Beam example project

Stateflow ⭐ 9

Prototype which extracts stateful dataflows by analysing Python code.

Example Beam ⭐ 8

Playground for Apache Beam and Scio experiments, driven by real-world use cases.

Hedera Etl ⭐ 8

ETL scripts for Hedera Hashgraph

Beam Postgres ⭐ 8

Light IO transforms for Postgres read/write in Apache Beam pipelines.

Apache Beam Io Extras ⭐ 6

The missing I/O PTransforms of Apache Beam in python; which already exist in Java SDK based but not yet supported in the official apache-beam module.

Stream Processing ⭐ 6

Learn how to develop and test stateful streaming and batch data pipelines

Apache Beam Internals ⭐ 6

The Internals of Apache Beam

Fluent Tfx ⭐ 5

A fluent API layer for tensorflow extended e2e machine learning pipelines

1-45 of 45 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.