Kafka Message Scheduler

Scheduler for low-frequency and long-term scheduling of delayed messages to Kafka topics.
Alternatives To Kafka Message Scheduler
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
3 days ago135apache-2.0Java
Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
Schema Registry1,930
16 hours ago257otherJava
Confluent Schema Registry for Kafka
a day ago1July 25, 201647mitJava
A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues
7 years ago19apache-2.0Scala
MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka.
2 years agoapache-2.0Java
Change Data Capture (CDC) service
3 days ago54apache-2.0HTML
Karapace - Your Apache Kafka® essentials in one tool
Kafka Connect Oracle237
2 years ago57apache-2.0Java
Kafka Source Connector For Oracle
Abris19555 months ago17October 06, 20209apache-2.0Scala
Avro SerDe for Apache Spark structured APIs.
Srclient1911610 days ago27June 03, 20222apache-2.0Go
Golang Client for Schema Registry
2 years ago18apache-2.0Shell
Collection of Kubernetes Operators built with KUDO.
Alternatives To Kafka Message Scheduler
Select To Compare

Alternative Project Comparisons

Kafka Message Scheduler

Build Status Docker Image Version (latest by date) Docker Pulls

This application is a scheduler for low-frequency and long-term scheduling of delayed messages to Kafka topics.


This component was initially designed for Sky's metadata ingestion pipeline. We wanted to manage content expiry (for scheduled airings or on-demand assets) in one single component, instead of implementing the expiry logic on all consumers.

Given that the pipeline is based on Kafka, it felt natural to use it as input, output and data store.

How it works

The Kafka Message Scheduler (KMS for short) consumes messages from configured source (schedule) topics. On this topic:

  • message keys are "Schedule IDs" - string values, with an expectation of uniqueness
  • message values are Schedule messages, encoded in Avro binary format according to the Schema.

A schedule is composed of:

  • The topic you want to send the delayed message to
  • The timestamp telling when you want that message to be delivered
  • The actual message to be sent, both key and value

The KMS is responsible for sending the actual message to the specified topic at the specified time.


If the timestamp of when to deliver the message is in the past, the schedule will be sent immediately.

The Schedule ID can be used to delete a scheduled message, via a delete message (with a null message value) in the source topic.

Startup logic

When the KMS starts up it uses the kafka-topic-loader to consume all messages from the configured schedule-topics and populate the scheduling actors state. Once this has completed, all of the schedules loaded are scheduled and the application will start normal processing. This means that schedules that have been fired and tombstoned, but not compacted yet, will not be replayed during startup.


To generate the avro schema from the Schedule case class, run sbt schema. The schema will be written to avro/target/schemas/schedule.avsc.

How to run it

Start services

docker-compose pull && docker-compose up -d

Send messages

With the services running, you can send a message to the defined scheduler topic (scheduler in the example above). See the Schema section for details of generating the Avro schema to be used.


Metrics are exposed and reported using Kamon. By default, the Kamon Prometheus reporter is used for reporting and the scraping endpoint for Prometheus is exposed on port 9095 (this is configurable by setting the PROMETHEUS_SCRAPING_ENDPOINT_PORT environment variable).

Prometheus is included as part of the docker-compose and will expose a monitoring dashboard on port 9090.

Topic configuration

The schedule-topics must be configured to use log compaction. This is for two reasons:

  1. to allow the scheduler to delete the schedule after it has been written to its destination topic.
  2. because the scheduler uses the schedule-topics to reconstruct its state - in case of a restart of the KMS, this ensures that schedules are not lost.

Recommended configuration

It is advised that the log compaction configuration of the schedule-topics is quite aggressive to keep the restart times low, see below for recommended configuration:

cleanup.policy: compact
delete.retention.ms: 3600000
min.compaction.lag.ms: 0
min.cleanable.dirty.ratio: "0.1"
segment.ms: 86400000
segment.bytes: 100000000


Until this issue is addressed the KMS does not fully support horizontal scaling. Multiple instances can be run, and Kafka will balance the partitions, however schedules are likely to be duplicated as when a rebalance happens the state for the rebalanced partition will not be removed from the original instance. If there is a desire to run multiple instances before that issue is addressed, it is best to not attempt dynamic scaling, but to start with your desired number of instances.

Popular Kafka Projects
Popular Schema Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.