Karafka

Ruby and Rails efficient Kafka processing framework
Alternatives To Karafka
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Karafka1,91522618 hours ago149December 07, 202346otherRuby
Ruby and Rails efficient Kafka processing framework
Racecar46562a month ago61October 18, 202326apache-2.0Ruby
Racecar: a simple framework for Kafka consumers in Ruby
Workshop Codecon Kafka11
3 years agon,ullmitJava
Exemplos do workshop realizado na codecon.dev sobre arquitetura event-driven com Apache Kafka.
Kafka_command8
5 years ago3February 15, 2019mitRuby
A simple Kafka management UI
Stackbox7
2 years ago25mitRuby
Create app stacks loaded with all your favourite clients, services and infra along with code boilerplates in under 5 mins.
Messaging Adapter5
5 years ago4October 18, 2018mitRuby
Messaging adapter gem for Ruby, useful for using message brokers (RabbitMQ, Kafka,...) in Ruby applications
Druiddb Ruby4
6 years ago1mitRuby
Ruby client for Druid using the Kafka Indexing Service
Eventsourcer3
6 years ago3mitRuby
A helper library for event sourcing Rails ActiveRecord model changes
Action_monitor2
5 years agomitRuby
a Rails model monitor, which monitor model changes and translate these changes to logs, kafka and so on.
Microservice Architecture2
4 months ago32mitRuby
Este projeto é um exemplo de uma solução de "e-commerce" usando padrões de arquitetura, com 2 principais, que são: SAGA with Choreography e Event Sourcing.
Alternatives To Karafka
Select To Compare


Alternative Project Comparisons
Readme

karafka logo

Build Status Gem Version Join the chat at https://slack.karafka.io

About Karafka

Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:

# Define what topics you want to consume with which consumers in karafka.rb
Karafka::App.routes.draw do
  topic 'system_events' do
    consumer EventsConsumer
  end
end

# And create your consumers, within which your messages will be processed
class EventsConsumer < ApplicationConsumer
  # Example that utilizes ActiveRecord#insert_all and Karafka batch processing
  def consume
    # Store all of the incoming Kafka events locally in an efficient way
    Event.insert_all messages.payloads
  end
end

Karafka uses threads to handle many messages simultaneously in the same process. It does not require Rails but will integrate tightly with any Ruby on Rails applications to make event processing dead simple.

Getting started

karafka web ui

If you're entirely new to the subject, you can start with our "Kafka on Rails" articles series, which will get you up and running with the terminology and basic ideas behind using Kafka:

If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to visit our Getting started guides and the example apps repository.

We also maintain many integration specs illustrating various use-cases and features of the framework.

TL;DR (1 minute from setup to publishing and consuming messages)

Prerequisites: Kafka running. You can start it by following instructions from here.

  1. Add and install Karafka:
# Make sure to install Karafka 2.3
bundle add karafka --version ">= 2.3.0"

bundle exec karafka install
  1. Dispatch a message to the example topic using the Rails or Ruby console:
Karafka.producer.produce_sync(topic: 'example', payload: { 'ping' => 'pong' }.to_json)
  1. Run Karafka server and see the consumption magic happen:
bundle exec karafka server

[86d47f0b92f7] Polled 1 message in 1000ms
[3732873c8a74] Consume job for ExampleConsumer on example started
{"ping"=>"pong"}
[3732873c8a74] Consume job for ExampleConsumer on example finished in 0ms

Want to Upgrade? LGPL is not for you? Want to help?

I also sell Karafka Pro subscriptions. It includes a commercial-friendly license, priority support, architecture consultations, enhanced Web UI and high throughput data processing-related features (virtual partitions, long-running jobs, and more).

10% of the income will be distributed back to other OSS projects that Karafka uses under the hood.

Help me provide high-quality open-source software. Please see the Karafka homepage for more details.

Support

Karafka has Wiki pages for almost everything and a pretty decent FAQ. It covers the installation, setup, and deployment, along with other useful details on how to run Karafka.

If you have questions about using Karafka, feel free to join our Slack channel.

Karafka has priority support for technical and architectural questions that is part of the Karafka Pro subscription.

Popular Kafka Projects
Popular Ruby On Rails Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Ruby
Ruby On Rails
Apache
Kafka
Wiki
Apache Kafka
Background Jobs
Kafka Consumer
Kafka Producer
Kafka Client