Awesome Open Source
Awesome Open Source

KQ: Kafka Job Queue for Python

Build CodeQL codecov PyPI version GitHub license Python version

KQ (Kafka Queue) is a lightweight Python library which lets you enqueue and execute jobs asynchronously using Apache Kafka. It uses kafka-python under the hood.


  • Support for Python 3.5 will be dropped from KQ version 3.0.0.
  • See releases for latest updates.



Install using pip:

pip install kq

Getting Started

Start your Kafka instance. Example using Docker:

docker run -p 9092:9092 -e ADV_HOST= lensesio/fast-data-dev

Define your KQ module:

import logging

from kafka import KafkaConsumer
from kq import Worker

# Set up logging.
formatter = logging.Formatter("[%(levelname)s] %(message)s")
stream_handler = logging.StreamHandler()
logger = logging.getLogger("kq.worker")

# Set up a Kafka consumer.
consumer = KafkaConsumer(

# Set up a worker.
worker = Worker(topic="topic", consumer=consumer)

Start your worker:

[INFO] Starting Worker(hosts= topic=topic, group=group) ...

Enqueue a function call:

import requests

from kafka import KafkaProducer
from kq import Queue

# Set up a Kafka producer.
producer = KafkaProducer(bootstrap_servers="")

# Set up a queue.
queue = Queue(topic="topic", producer=producer)

# Enqueue a function call.
job = queue.enqueue(requests.get, "")

# You can also specify the job timeout, Kafka message key and partition.
job = queue.using(timeout=5, key=b"foo", partition=0).enqueue(requests.get, "")

The worker executes the job in the background:

[INFO] Starting Worker(hosts=, topic=topic, group=group) ...
[INFO] Processing Message(topic=topic, partition=0, offset=0) ...
[INFO] Executing job c7bf2359: requests.api.get("")
[INFO] Job c7bf2359 returned: <Response [200]>

See the documentation for more information.

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
python (50,968
python3 (1,530
kafka (397
apache-kafka (38
kafka-client (31
job-queue (23
kafka-producer (18
kafka-consumer (17

Find Open Source By Browsing 7,000 Topics Across 59 Categories