Awesome Open Source

Programming Languages

Search results for scala avro

109 search results found

Upserts, Deletes And Incremental Processing on Big Data.

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

Kafka Storm Starter ⭐ 729

[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Avro schema generation and serialization / deserialization for Scala

Bijection ⭐ 658

Reversible conversions between types

Sparklearning ⭐ 573

Learning Apache spark,including code and data .Most part can run local.

Spark Avro ⭐ 535

Avro Data Source for Apache Spark

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka.

Ratatool ⭐ 333

A tool for data sampling, data generation, and data diffing

Mu Haskell ⭐ 311

Mu (μ) is a purely functional framework for building micro services.

Avro SerDe for Apache Spark structured APIs.

Trifecta ⭐ 211

Trifecta is a web-based and CLI tool that simplifies inspecting Kafka messages and Zookeeper data. Additionally, the CLI tool provides the capability to import/export data to/from ElasticSearch and MongoDB.

Avrohugger ⭐ 198

Generate Scala case class definitions from Avro schemas

Magnolify ⭐ 155

A collection of Magnolia add-on modules

Bigdata Playground ⭐ 154

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

Scala library to eliminate boilerplate

Spark Bigquery ⭐ 149

Google BigQuery support for Spark, SQL, and DataFrames

Eel Sdk ⭐ 140

Big Data Toolkit for the JVM

Coulomb ⭐ 139

coulomb: unit analysis for Scala

Sbt Avrohugger ⭐ 129

sbt plugin for generating Scala sources for Apache Avro schemas and protocols.

Kafka Serialization ⭐ 121

Lego bricks to build Apache Kafka serializers and deserializers

Scalavro ⭐ 119

A reflection-based Avro library in Scala.

Avro2tf ⭐ 117

Avro2TF is designed to fill the gap of making users' training data ready to be consumed by deep learning training frameworks.

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

Kafka Connect Twitter ⭐ 75

Kafka Connect Sink/Source for Twitter

Gcs Tools ⭐ 70

GCS support for avro-tools, parquet-tools and protobuf

Shapeless Datatype ⭐ 67

Shapeless utilities for common data types

Spark Structured Streaming ⭐ 66

Spark structured streaming with Kafka data source and writing to Cassandra

Avro Parquet Spark Example ⭐ 61

An example of using Avro and Parquet in Spark SQL

Avro Scala Macro Annotations ⭐ 57

Compile-time tools for working with Avros in Scala

Kafka Serde Scala ⭐ 56

Implicitly converts typeclass encoders to kafka Serializer, Deserializer, Serde.

Scalable Consistency Adjustable Data Storage

Kafka Scala Examples ⭐ 42

Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala

Standalone alternatives to Kafka Connect Connectors

Etl Light ⭐ 38

A light Kafka to HDFS/S3 ETL library based on Apache Spark

Simplesparkavroapp ⭐ 32

Simple Spark app that reads and writes Avro data

Kafka Message Scheduler ⭐ 32

Scheduler for low-frequency and long-term scheduling of delayed messages to Kafka topics.

Akka Serialization Test ⭐ 31

Study on akka-serialization using ProtoBuf, Kryo and Avro

Fast Avro Write ⭐ 30

Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.

Xml Avro ⭐ 30

Convert XSD -> AVSC and XML -> AVRO

Parquet Extra ⭐ 29

A collection of Apache Parquet add-on modules

Sbt Avro ⭐ 28

sbt plugin for compiling Avro schemas, similar to sbt-protobuf

Recursion Schemes Lc2018 ⭐ 24

Rrecursion schemes without the barbed wire

Affinity ⭐ 24

Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Scala Kafka Twitter ⭐ 23

Example integration of Kafka, Avro & Spark-Streaming on live Twitter feed

Salat Avro ⭐ 23

Fast bi-directional Scala case class to Avro serialization

Avro Sql ⭐ 23

Use SQL to transform your avro schema/records

Avro Schema Evolution made easy

An SBT plugin for automatically calling Avro code generation and a thin scala wrapper for reading and writing Avro files

Data Model Generator ⭐ 20

Data model generator based on Scala case classes

Dendrites ⭐ 20

Scala library for Reactive streaming Microservices, CQRS, Event Sourcing, Event Logging, & message-driven apps.

Sbt Avro4s ⭐ 20

Sbt plugin for avro4s

Etl Starter Kit ⭐ 18

📁 Extract, Transform, Load (ETL) 👷 refers to a process in database usage and especially in data warehousing. This repository contains a starter kit featuring ETL related work.

Akka Avro Evolution ⭐ 17

Demonstrate schema-less serialization and schema evolution with Akka Persistence and Avro

Confluent Spark Avro ⭐ 17

Spark UDFs to deserialize Avro messages with schemas stored in Schema Registry.

Spark Kafka Avro ⭐ 17

POC: Spark consumer for bottledwater-pg Kafka Avro topics

Salesforce2hadoop ⭐ 16

Import Salesforce data into Hadoop HDFS in Avro format

Spark Bigquery ⭐ 15

Google BigQuery data source for Apache Spark

Pure functional wrapper for RocksDB

Sbt Avro ⭐ 15

Plugin SBT to Generate Scala classes from Apache Avro schemas hosted on a remote Confluent Schema Registry.

Spark Dbf ⭐ 14

Spark SQL DBF Library

Kinesis Tee ⭐ 12

Unix tee, but for Kinesis streams

Gradle Avrohugger Plugin ⭐ 12

Gradle plugin for generating scala case classes from apache avro schemas, datafiles and protocols

Spark Streaming Kafka Example ⭐ 11

An example project using Spark Streaming with Kafka message and Avro serialization

Parquet Custom Reader Writer ⭐ 11

Simple implementation of a custom parquet reader/writer

Serialization Benchmark ⭐ 10

benchmark for modern serialization systems: Apache Avro, Protocol Buffers, Apache Thrift and MessagePack written in Scala

Zio Kafka Streams ⭐ 10

Kafka Streams for Scala with ZIO

Confluent Platform Spark Streaming ⭐ 10

Working example of consuming Avro data from Kafka with Spark Streaming

Amazon S3 Tagging Spark Util ⭐ 10

A small project to allow publishing data to Apache Kafka, Apache Pulsar or any other target system

Typebook ⭐ 10

An Avro Schema Registry with MySQL backend.

Utils Open ⭐ 10

Open source utilties

Scala case classes over 0mq, salat-avro for serialization, akka & lift actor integration

Avrohugger Maven Plugin ⭐ 9

Maven plugin for generating Scala case classes and ADTs from Apache Avro schemas, datafiles, and protocols

Spark Avro Example ⭐ 9

Example of using Avro with Apache Spark

Some Avro operations in Scala

Scabillmatch ⭐ 9

Policy diffusion in the US legislature

Cassandra Summit Demo ⭐ 9

Hadoop integration demo for the Cassandra Summit

Cloudera Cca175 ⭐ 8

CCA Spark and Hadoop Developer Certification

Cloud Storage Extension ⭐ 8

Exasol Cloud Storage Extension for accessing formatted data Avro, Orc and Parquet, on public cloud storage systems

Wsdl2avro ⭐ 7

WSDL2Avro is a Scala library that lets you convert datatypes from a SOAP WSDL to Avro Schemas.

Flink Avro State Serialization ⭐ 7

Scio Contrib ⭐ 7

Community-supported add-ons for Scio

big data query console command and script for scala

Metrifier ⭐ 7

Comparing HTTP endpoints against mu-rpc services.

Avro Codegen ⭐ 7

Scala code generator for Avro schemas.

Avro4s Schema Registry Kafka Streams Demo ⭐ 7

Demo application for Avro4s/Schema Registry/Kafka Streams integration

Copy_to_postgres ⭐ 7

Command line tool for fast importing to Postgres. Reads Avro, Json, Csv and provides support for simple mappings.

Blockchain Importer ⭐ 6

Functional, typesafe, well-tested and composable streaming blockchain importer, drop-in solution for Kafka ecosystem.

Avro Scala Demo ⭐ 6

Demonstration of using avro4s library in Scala to provide basic Avro serialization

Kudu From Avro ⭐ 6

A small Command Line tool to create an Kudu table from an Avro schema or from SQL script

Schemakeeper ⭐ 6

Schemakeeper - yet another schema registry for Avro, Thrift and Protobuf schemas

Pipelines Model Serving ⭐ 6

Implementation of Model serving in pipelines

Flink Avro State Recovery ⭐ 6

Example project on how to do state recovery in Apache Flink using Apache Avro

Marley is a library for serialising Scrooge generated classes representing Thrift objects as Avro with minimal boilerplate.

Akkastreams Kafka Example ⭐ 6

An example project using Akka Streams with Kafka and a Schema Registry to read and write messages in Avro format.

Kafkaavroexamples ⭐ 6

Set of repositories where we use Apache Avro with Apache Kafka (producer/consumer/stream/scheme registry)

Related Searches

Scala Sbt (4,178)

Scala Spark (3,279)

Scala Akka (2,120)

Java Scala (1,794)

Scala Play Framework (1,309)

Plugin Scala (1,079)

Scala Kafka (969)

Scala Functional Programming (942)

Scala Scalajs (887)

Docker Scala (728)

1-100 of 109 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.