Dropwizard TechEmpower Benchmark - Part 1: Let’s Begin
Some of you may have heard of the TechEmpower Web Framework Benchmarks. As it turns out, the open source project that I’m involved with (Dropwizard) is featured in the benchmark. Dropwizard hasn’t fared terribly well in the benchmark, however, which is surprising because I’ve seen it in production. It’s been pretty snappy.
Determined to get to the bottom of the results (maybe I should rephrase that), I’ve forked and locally cloned the benchmark repo, located Dropwizard in the frameworks/Java/dropwizard directory, and followed the README.md instructions to run the benchmark locally.
The results aren’t pretty. Granted, the benchmark runs inside a VirtualBox instance alongside the Dropwizard app and a MySQL database. It’s not the raw latency that concerns me so much as the standard deviation and the long tail latency (i.e. latency at the 99th percentile). These are typically indications of scaling issues. Even at 8 concurrent connections with 2 driving threads, long tail latency for a simple JSON endpoint is alarmingly high:
Running 15s test @ http://127.0.0.1:9090/json 2 threads and 8 connections Thread Stats Avg Stdev Max +/- Stdev Latency 1.08ms 2.36ms 55.03ms 95.36% Req/Sec 5.81k 626.98 7.28k 82.67% Latency Distribution 50% 602.00us 75% 732.00us 90% 1.20ms 99% 12.42ms 173528 requests in 15.01s, 27.47MB read Requests/sec: 11561.92
As you can see, 75% of requests finish in under 1 millisecond. Even at 90% we’re still finishing in just over 1 millisecond. The slowest 1% of requests, however, take more than 10 milliseconds. As the concurrency maxes out at 256 connections and 2 threads, the median response time is unnacceptably slow for such a basic endpoint (even for an underpowered virtual machine) and the 99th percentile is heading towards 1 second:
Running 15s test @ http://127.0.0.1:9090/json 2 threads and 256 connections Thread Stats Avg Stdev Max +/- Stdev Latency 74.24ms 130.63ms 1.28s 88.12% Req/Sec 5.45k 0.94k 8.24k 72.00% Latency Distribution 50% 15.04ms 75% 85.84ms 90% 233.41ms 99% 620.43ms 162873 requests in 15.01s, 25.78MB read Requests/sec: 10848.85
Equipped with the fabulous IntelliJ IDEA, the indispensable Java Mission Control, insatiable curiosity, and a desire to see Dropwizard move up the ranks a bit I’m going to dive into the benchmark, into Dropwizard, and into Dropwizard’s dependencies to see what I can learn and what I can improve. I plan to do a series of posts showing my methods and remedies in the hope that it proves useful or educational to others.
If that sounds interesting to you, you can follow me on Twitter for updates. I’ve already identified a few improvements, which I hope are as entertaining as they are educational.
