Deploy Flask with a Celery task queue and Flower dashboard using Kubernetes

In this article, we walkthrough the steps to deploy a simple Flask app integrated with Celery , an asynchronous task queue based on message passing. It makes uses RabbitMQ and/or Redis as the message broker and results backend, which can be cumbersome to provision and orchestrate. We use Kubernetes to ease the management of the various deployments and services required in our stack, which leaves us more time to focus on coding and less on DevOps.

Contents

2Deploy the RabbitMQ Message Broker 3Deploy the Redis Results Backend 4Deploy the Flask web app 4.1requirements.txt 4.5web-flask-deployment.yaml 4.6web-flask-service.yaml 5Deploy the Redis Queue (RQ) workers 5.2rq-worker-deployment.yaml 7RQ Dashboard (Optional) 7.1requirements.txt 7.3rq-dashboard-deployment.yaml 7.4rq-dashboard-service.yaml 7.5Alternative Solution: Integrate the Dashboard in our Flask app

This guide assumes some familiarity with Flask integrated with Celery and focuses more on provisioning the various components using Docker and Kubernetes.

The post Using Celery With Flask by @miguelgrinberg and the Celery Based Background Tasks page in the Flask official docs provide good starting points and should get you up to speed fairly quickly.

Let's start by creating and navigating to the directory to contain artifacts relevant to this deployment:

$ mkdir flask-celery $ cd flask-celery

2Deploy the RabbitMQ Message Broker

https://hub.docker.com/_/rabbitmq/

$ kubectl run rabbitmq --image=rabbitmq:3.6.2-management --replicas=1 --port=5672 \ > --dry-run --output=yaml > rabbitmq-deployment.yaml $ kubectl create -f rabbitmq-deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: creationTimestamp: null labels: run: rabbitmq name: rabbitmq spec: replicas: 1 selector: matchLabels: run: rabbitmq strategy: {} template: metadata: creationTimestamp: null labels: run: rabbitmq spec: containers: - image: rabbitmq:3.6.2-management name: rabbitmq ports: - containerPort: 5672 - containerPort: 15672 resources: {} status: {}

$ kubectl expose deployment rabbitmq --type=NodePort \ > --dry-run --output=yaml > rabbitmq-service.yaml $ kubectl create -f rabbitmq-service.yaml

3Deploy the Redis Results Backend

https://hub.docker.com/_/redis/

$ kubectl run redis-server --image=redis --replicas=1 --port=6379 \ > --dry-run --output=yaml > redis-server-deployment.yaml $ kubectl create -f redis-server-deployment.yaml $ kubectl expose deployment redis-server --dry-run --output=yaml > redis-server-service.yaml $ kubectl create -f redis-server-service.yaml

4Deploy the Flask web app