Hi, I'm trying to build up a different kind of web server framework benchmark, where measurement is not on peak performance, but on capacity and latency. That is, the numbers we keep stable are:
* 10k connections * 30k req/s * 2053 bytes of static content served by a GET request on a simple route * Near perfect 1 gigabit connection between two machines * The test runs for 20 minutes. This is in contrast to many such measurements. The recent "round 10" TechEmpower benchmark will for instance let connections vary, try to push as many req/s through and report the maximal number for a "Hello World!" message. So for one system, they will report the number for 1024 connections, whereas another system will have 4096 connections, whichever has the highest peak performance. This is a big no-no in my book. Furthermore, I'm questioning the correctness of their latency measurements because they report average latency and std. deviation without establishing that their distributions are normal. And they don't report the median latency so we can see they aren't. The thing I'm interested in is latency of the 30k req/s, run over a 20 minute period. That is, employing Gil Tene's HDRHistogram, we can accurately measure latencies for each request and make a distribution function plot of those latencies. I'm sitting with approximately 15 frameworks, spread out over multitiple languages. The numbers are vastly different from typical benchmarks and they are interesting to report. For Clojure, I started out with a simple adaptation of the Compojure benchmark from the TechEmpower-land. But it exhibits lots of queueing when trying to process requests simply due to maximizing the CPU cores of the System-under-test. My JVM-fu is quite weak, which is a big problem in these tests. I more or less need a setup, which has some kind of well-tunedness in advance, because I have little hope in achieving it myself on my own. It's somewhat like 10 years ago I last tuned GC on a JVM and I bet things change. Currently Clojure is in the ballpark of Python and Ruby in speed, which is so slow I'm going to argue that there is something wrong with my test implementation: http://imgur.com/wCFnFnd (X-axis are percentiles, note it's compression toward the 0th percentile. I'm interested in the upper percentiles) The graphs of Python, Ruby and Clojure/Compojure here are clear indicators that queueing/stalling is going on even at the very low percentiles. The solutions Go and Java/Undertow are in the graph for comparison with a couple of frameworks which does not exhibit queueing[0]. The highest scoring benchmark in the techempower rounds is a combination of Clojure and Resin (whatever resin is, I've not really dug too much into it), but I'm told that http-kit may be a more viable route. Also, I might be needing some underlying web server I guess, and what are good suggestions there?. So my question is this: "If you were to write a webserver where curl http://localhost:8080/ would return the first 2053 bytes of "Alice in Wonderland"[1], what would you do?" In particular if the above data is known: capacity is 10k connections, rates are 30k req/s. Goal is low-latency operation. As I said, the current numbers are just too far off, but in contrast to Ruby/Python, I think Clojure might just be my bad implementation of what is going on. That is, if you went tabula rasa on my frankenstein monster, what would you do? [0] What is not present on this graph is the bi-modal behavior of Go and Undertow once GC kicks in, however. [1] This is a grave mistake! It should have been "Jabberwocky!" The techempower code is here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Clojure/compojure and my field-surgery is below for hello.clj: (ns hello.handler (:import com.mchange.v2.c3p0.ComboPooledDataSource) (:use compojure.core ring.middleware.json ring.util.response) (:require [compojure.handler :as handler] [compojure.route :as route])) (def plaintext "Test 6: Plaintext" {:status 200 :headers {"Content-Type" "text/plain; charset=utf-8"} :body "CHAPTER I. Down the Rabbit-Hole Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, <and what is the use of a book,> thought Alice <without pictures or conversations?> So she was considering in her own mind (as well as she could, for the hot day made her feel very sleepy and stupid), whether the pleasure of making a daisy-chain would be worth the trouble of getting up and picking the daisies, when suddenly a White Rabbit with pink eyes ran close by her. There was nothing so very remarkable in that; nor did Alice think it so very much out of the way to hear the Rabbit say to itself, <Oh dear! Oh dear! I shall be late!> (when she thought it over afterwards, it occurred to her that she ought to have wondered at this, but at the time it all seemed quite natural); but when the Rabbit actually took a watch out of its waistcoat-pocket, and looked at it, and then hurried on, Alice started to her feet, for it flashed across her mind that she had never before seen a rabbit with either a waistcoat-pocket, or a watch to take out of it, and burning with curiosity, she ran across the field after it, and fortunately was just in time to see it pop down a large rabbit-hole under the hedge. In another moment down went Alice after it, never once considering how in the world she was to get out again. The rabbit-hole went straight on like a tunnel for some way, and then dipped suddenly down, so suddenly that Alice had not a moment to think about stopping herself before she found herself falling down a very deep well. Either the well was very deep, or she fell very slowly, for she had plenty of time as she went down to look about her and to wonder what was going to happen next. First, she tried to look down and make out what she was coming to, but it was too dark to see anything; then she looked at the sides of the well, and noticed that they were filled with cupboards......"}) (defroutes app-routes (GET "/" [] plaintext) (route/not-found "Not Found")) (def app "Format responses as JSON" (wrap-json-response app-routes)) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
