Celery is an asynchronous task queue/job queue based on distributed message passing. We are big fans of Celery and it’s an important part of our stack. As with any distributed system, Celery comes with it’s own set of challenges. In this post, I specifically want to discuss how we use Celery to push webhooks to our customers and the race conditions caused because of using database transactions.
The Webhook Architecture
We generate alerts for delays on the way and push these events over webhooks. Depending on the event, different entities in the database get updated and an event object is stored in the db. We then put the event id on RabbitMQ where it is picked up by one of our worker nodes and pushed to the webhook URL.
The Transaction Race Condition
As you can imagine, all of these updates have to be done in a transaction to guarantee the database is in the correct state. This is what our code looked like:
Once deployed, we started noticing the following error, intermittently:
Error: Task events.jobs.push_event (837a1d29-4821-2b9e-844f-b8e5b40657): “DoesNotExist(‘Event matching query does not exist.’, )
The task was being picked up by the worker before the transaction was complete. Because our isolation level was set at READ COMMITTED , the event from the uncommitted transaction was not visible to the worker and hence the task was failing. We needed to put the event on the queue AFTER the transaction had been committed.
The Solution
Starting version 1.9, Django has introduced the on_commit hook for transactions. We canpass a function to this hook and it willbe called when the transaction is successfully committed. To be able to use this across different tasks, we use an abstract Celery task class that we call the TransactionAwareTask . Using this, we can ensure that our tasks arefired only after the transaction has been committed. Thisis what our code looks like now:
This solution is simple and generic enough that it can be used across a variety of tasks and it hasn’t let us down since.
Have questions? Suggestions? Join the discussion onslack.
Like what we are doing? Sign up to use HyperTrack and build location tracking features!