Note: We're finishing the new Karafka
2.0 but for now, please use
1.4. All the documentation presented here refers to
Framework used to simplify Apache Kafka based Ruby applications development.
# Define what topics you want to consume with which consumers Karafka::App.consumer_groups.draw do topic 'system_events' do consumer EventsConsumer end end # And create your consumers, within which your messages will be processed class EventsConsumer < ApplicationConsumer # Example that utilizes ActiveRecord#insert_all and Karafka batch processing def consume # Store all of the incoming Kafka events locally in an efficient way Event.insert_all params_batch.payloads end end
Karafka allows you to capture everything that happens in your systems in large scale, providing you with a seamless and stable core for consuming and processing this data, without having to focus on things that are not your business domain.
Karafka not only handles incoming messages but also provides tools for building complex data-flow applications that receive and send messages.
Karafka provides a higher-level abstraction that allows you to focus on your business logic development, instead of focusing on implementing lower level abstraction layers. It provides developers with a set of tools that are dedicated for building multi-topic applications similar to how Rails applications are being built.
Karafka based applications can be easily deployed to any type of infrastructure, including those based on:
If you have any questions about using Karafka, feel free to join our Gitter chat channel.
If you're completely new to the subject, you can start with our "Kafka on Rails" articles series, that will get you up and running with the terminology and basic ideas behind using Kafka:
If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:
git clone https://github.com/karafka/example-app ./example_app
then, just bundle install all the dependencies:
cd ./example_app bundle install
and follow the instructions from the example app Wiki.
Note: you need to ensure, that you have Kafka up and running and you need to configure Kafka seed_brokers in the
If you need more details and know how on how to start Karafka with a clean installation, read the Getting started page section of our Wiki.
First, thank you for considering contributing to the Karafka ecosystem! It's people like you that make the open source community such a great community!
Each pull request must pass all the RSpec specs, integration tests and meet our quality requirements.
Fork it, update and wait for the Github Actions results.