KQ: Simple Job Queue for Python Using Kafka

KQ: Kafka-based Job Queue for python

KQ (Kafka Queue) is a light-weight Python library which provides a simple API to queue and process jobs asynchronously in the background. It is backed by Apache Kafka and designed primarily for ease of use.

Requirements Apache Kafka 0.9+ Python 2.7+ 3.4+ or 3.5+ Getting Started

First, ensure that your Kafka instance is up and running:

# This command is just an example ~$ ./kafka-server-start.sh -daemon server.properties

Let's say you want to run the following function asynchronously:

import time def my_func(foo, bar, baz=None): """This is a blocking function.""" time.sleep(10) return foo, bar, baz

Start a KQ worker:

~$ kq worker --verbose [INFO] Starting Worker(topic=foobar) ...

Enqueue the function call as a job:

# Import the blocking function from my_module import my_func # Initialize a queue from kq import Queue q = Queue() # Enqueue the function call q.enqueue(my_func, 1, 2, baz=3)

Sit back and watch the worker process the job in the background:

~$ kq worker --verbose [INFO] Starting Worker(topic=default) ... [INFO] Processing Record(topic=default, partition=5, offset=3) ... [INFO] Running Job 1b92xle0: my_module.my_func(1, 2, baz=3) ... [INFO] Job 1b92xle0 returned: (1, 2, 3)

Check out the full documentation for more details!

Installation

To install a stable version from PyPI (recommended):

~$ pip install kq

To install the latest version directly fromGitHub:

~$ pip install -e git+git@github.com:joowani/kq.git@master#egg=kq

You may need to use sudo depending on your environment setup.

Credits

This project was inspired byRQ and built on top ofkafka-python.