(Version 1.0)
Benchmarking & Comparing Open-Source Load Testing ToolsRagnar Lnn
There are tons of load testing tools, both open- and closed-source. Open-source tools are growing in popularity, and we use mainly open-source software (OSS) at Load Impact, so we thought it might be useful to take a deep look at the available OSS load testing options.
OSS load testing options represent kind of a jungle, and the difference between one tool and the next in terms of usability, performance, feature set or reliability can be enormous. Thus, we’d like to help people find the right tool for their use case.
We have chosen to look at what we consider to be the most popular, open-source load testing tools out there today. This list includes:
Jmeter Gatling Locust The Grinder Apachebench Artillery Tsung Vegeta Siege Boom Wrk
This review contains both my own personal views on the good and bad sides of these load testing tools, and my thoughts based on a round of benchmark testing that gives a sense about relative tool performance.
If you are mainly interested in the results from the benchmarking, they will be posted in a followup article very soon :)
So ― what's the setup?
Setting up the Open Source Load Testing Review
We installed, configured and ran these tools from the command line, plus spent a lot of time trying to extract results from them using shell script and standard Unix tools. Then we have tried to figure out the relative performance of the tools. First, by manually trying to squeeze the best performance out of each one, optimizing configuration parameters for the tool in question, and then by running a benchmark test on all the tools, trying to run them with as similar configuration as possible.
The benchmark numbers are one thing, but the opinions on usability that I give will be colored by my use case. For example, a tool that is hard to run from the command line, or that is hard to get useful results from when you run it that way, will make me frustrated. And when I’m frustrated, I’ll have to whine and complain about it. So that is what this article is all about.
One big caveat: I have not been looking much at the data visualization options included with each tool, so you may want to see this comparison as most useful in an automation setting, or if you are planning to push results into some external data storage and/or visualization system anyway.
Try it yourself: run the Docker image
To make things easy for people, I’ve created a public Docker image that can be used to easily repeat all the tests we have made (or to just run one of the load testing tools without having to install it yourself). Try this:
<span style="font-family: 'courier new', courier;"><i><span style="font-weight: 400;"> docker run loadimpact/loadgentest</span></i></span>
Or, if you want to be able to simulate extra network delay you should do:
docker run --cap-add=NET_ADMIN loadimpact/loadgentest
You can also build the Docker image yourself, if you clone our Github repo:
git clone https:// github.com/loadimpact/loadgentest
At https://github.com/loadimpact/loadgentest you will also find some documentation on how to use the setup, by the way!
So ... Which tool is best?To go against all common wisdom regarding how to keep visitors reading your article, I’ll go ahead and just say: Gatling! Which is funny, because a month ago I would never have said that, given that Gatling is kind of a modern Jmeter, and I usually hate Java applications.
So Gatling is number one, the best, overall winner, etc. Of course... if you’re into python you may be better off with Locust. Oh yeah, and if you wanthigh performance, Wrk would be a better option. However, Wrk is scriptable in Lua, not Python, and has a callback-based API that is not as nice as Locust’s.
If you just need to hit a single URL and nothing else, Apachebench may be the way to go. With Apachebench, you won’t get distracted by all its features because it can literally not do anything but hit one specific URL.
Actually, I think I’ll withdraw that single recommendation. Instead I will say which tools I think have something going for them, and why, and then I’ll show you some benchmarking figures and let you decide for yourself, because one size really doesn’t fit all here.
The top 3open source load testing tools
Gatling ( http://gatling.io )
Apart from maybe its text output to stdout (which is as messy as Locust’s), Gatling is a very nice load testing tool. Its performance is not fantastic, but good enough. It has a consistent design where things actually make a little bit of sense, and the documentation is very good.
It has a DSL (domain specific language) more or less equal to what Jmeter and Tsung offer. However, while Jmeter and Tsung use XML with their specific tags to implement stuff like loops, Gatling lets you define Scala classes that offer similar functionality but that are a lot more readable.
Yes, Scala . Apparently everyone reacts the same way upon hearing that. Gatling’s documentation specifically tells you not to panic, which of course made me panic, but once I realized I couldn’t get out of this and bothered to read the docs and look at the examples I saw that it’s not tricky at all.
My initial assumption was that Gatling would be as clunky to use as Jmeter since it’s a Java app and with DSL functionality similar to Jmeter. But I have now seen the error of my ways and while I still very much prefer to use a “real,” dynamic scripting language to define what happens in a load test, Gatling’s DSL is perhaps the second best (to a real, dynamic language) thing around. Here is how it can look:
import io.gatling.core.Predef._import io.gatling.http.Predef._
import scala.concurrent.duration._
class GatlingSimulation extends Simulation {
val httpConf = http
.baseURL("http://myhost.mydomain.com")
.disableCaching
.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8