Awesome Open Source
Awesome Open Source

kspp

Join the chat at https://gitter.im/kspp/Lobby

A high performance / realtime C++17 stream-processing framework with avro support. The design is influenced by apache kafka streams library. Change data capture for a postgres and sql server. Export to kafka, mqtt, postgres, elastic search, influxdb and avrofiles

Sources:

  • kafka (uses librdkafka)
  • kafka grpc proxy
  • aws kinesis (experimental, using aws sdk)
  • postgres (uses libpq)
  • microsoft sqlserver (uses freetds)
  • memory stream

Offset storage:

  • kafka
  • file
  • S3 (uses aws sdk)

Sinks:

  • kafka (using librdkafka)
  • mqtt (using paho libraries)
  • postgres (uses libpq)
  • influxdb
  • elastic search
  • files (avro)
  • S3 (avro)
  • memory stream

Statestores:

  • rocksdb
  • memory

Codecs:

  • avro (with confluent schema registry or grpc proxy)
  • text
  • json

Metrics:

  • prometheus

It is intended to be run in kubernetes but works equally well standalone

Platforms: Linux (Windows and Mac build are outdated)

Ubuntu 18.04 x64:

Remove stuff that you should not have...

sudo apt-get purge libprotobuf-dev libgrpc++-dev protobuf-compiler
sudo apt-get purge openssl-dev

Install build tools


sudo apt-get install -y software-properties-common
sudo apt-get update
sudo apt-get install -y g++ sudo pax-utils automake autogen shtool libtool git wget cmake unzip build-essential pkg-config sed bison flex

Install build deps

sudo apt-get install -y python-dev autotools-dev libicu-dev zlib1g-dev openssl libssl-dev libbz2-dev libsnappy-dev liblz4-dev libzstd-dev \
    libgoogle-glog-dev libgflags-dev libcurl4-openssl-dev libc-ares-dev liblzma-dev libpq-dev freetds-dev libxml2-dev \
    libfmt-dev libpcre2-dev libhttp-parser-dev 

install 3rd party deps we need to build from source (you can hack versions - see the file)

./3rdparty_install.sh 

build kspp

mkdir build && cd build
cmake  -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DLINK_SHARED=ON ..
make -j "$(getconf _NPROCESSORS_ONLN)"
sudo make install
cd ..

build docker image / ubuntu 18.04

cd docker-ubuntu
./build_3rdparty.sh
./build.sh

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
c-plus-plus (18,765
cpp (1,336
kafka (424
cpp17 (219
stream-processing (80
kafka-streams (39
kafka-connect (38