Async Through the Looking Glass

Async Through the LookingGlass Adventures in pythonLand

A lot of talk has been happening in the python community about asyncio. Asynchronous programming is the new hot thing in python. Inmy last post I talked about how to do asynchronous programming and what it is, but in this post we will cover whether or not it’s worth even bothering with this whole async thing. Let’s talk about some common use cases in python, and decide whether asyncio will serve you well.

Disclaimer: every point in this article assumes Python as the programming language. If you are using another language, some of these points might not be completely accurate.

The Pool ofTears

The first thing you need to understand is that python has a Global Interpreter Lock . This basically means that the python interpreter uses a lock to make sure only one thread is executing code at any given time. In other words, python multi-threaded applications still only use one CPU core, and are not any faster than standard single-threaded applications. In fact, in cpu-bound cases, a multi-threaded application can take longer than a single threaded one because of thread-contention. (Proof at bottom of the post).

This means that a single asynchronous thread will inherently have the same performance as a multi-threaded app, except for the fact that the async version wont have any thread contention.

In order to do true parallelism in python you need to use multiple processes. Each process in python has its own GIL and they do not share resources at all. If you want to maximize your CPU cores, typically the rule is you want your number of processes to equal the number of cores you have plus one. This rule applies to both threads and async programming.

The Mock Turtle’sStory

There is no silver bullet in programming, and asynchronous programming is no exception. There are certain cases where asynchronous programming might benefit you, and several others where it wont. There are even some areas where async could hurt you. In order to know if async will provide benefits, we have to consider the use cases. The most common use case would be a web-application backend (web server). This is one of the most common types of programming I come across in python, so I will try to cover it in depth. If you don’t deal with web programming, feel free to skip to the Queens Croquet Ground section.

Remember the days before online banking, when we would actually go to a bank, stand in line, and wait for them to accept our money? Since you might be too young to remember such dark days, here is a brief sum up of how banks work. It’s a cold day, and you’re in a hurry to cash your check. You walk into the bank and see a line of people standing in a single queue. You notice that there are several tellers helping customers. You wait in line, getting antsy because you left your car outside with the engine on. You get anxious when see a slow troublesome customer over at teller #4. Hoping that everyone in front of you only needs to simply cash a check, you stay in line. You know that teller #4 wont be freed up anytime soon, so you start cheering in your head for teller #2 to work quickly. When you finally get up to the teller, you hand them your check, and get the cash equivalent. How long you waited in line was dependent on how long it took the bankers to handle the people in front of you.

The fast food industry however has learned the art of maximizing throughput. After all, a synonym for speed is in the name. When you go to a fast food place, there is usually only one person taking orders. The person takes the order, you complete your transaction, then you wait for your food to arrive. The difference here is that while you are waiting for you food, the same teller that took your order is now taking the order of everyone else in line. Because you only wait to order, you go through the line a lot quicker.

Lobster Quadrille

The bank example is a synchronous web server. A typical python web server works by having thousands of threads (the tellers) ready to handle requests. The teller completes the entire transaction himself. If the teller has to wait on a manager or someone to come over, he will wait with you rather than moving to another customer. Just like the bank, your webserver can not handle more requests (customers) at a time than threads (tellers) you have in your pool (bank). However, because python can only have one thread running at a time, imagine that it’s more like the bank has only one cash box, and it can only accommodate one teller at a time. No two tellers can actually be counting cash (their main task) in parallel.

The fast-food example is the asynchronous example. In this example there is only one clerk accepting incoming requests as fast as possible. Since you are limited to a single cash-register, it makes sense to only have a single person (thread) handling requests. This way you don’t waste time “waiting for the cash register” (cpu scheduling). The fast-food place instead has waiting points; waiting for another person to complete their part of the order. When a wait happens, the employee can help another customer while waiting. For example imagine the person assembling your food as the database. While you are waiting for your food, the clerk is accepting new orders. But once your food is ready, the clerk puts it all together and hands it to you.

The astute reader might realize that the overall time waiting “to be finished” is still the same in both scenarios, the only difference is that in the fast-food example, you are waiting twice instead of all at once. This is true in the naive example, but your average wait times decrease as your tasks become denormalized. Imagine every person ordered the same meal at the restaurant. If this was the case, the bank style and the fast-food style of you getting your food would yield the same results. The reason why fast-food chains prefer an asynchronous flow is because it yields better results with varying sizes of orders. Imagine it’s dinner time, and everyone is ordering with their families, but all you want to do is order a shake. In a synchronous example, you have to wait for a family’s meal to be ready before you can even place your order. But in the asynchronous example, you get to place your order right away, and because it doesn’t require the food assemblers (database), you can get your shake right away, rather than waiting on others. This increases your overall throughput and lowers average response times. It also means that large tasks don’t make other non-large tasks more latent.

One thing I didn’t discuss is the cost of context switching with threads. I wont waste too much time discussing context switching, as I have already done so but needless to say, asynchronous web servers are theoretically faster than threaded web servers even in the naive case, because you have n