Wow! Thanks for all of this work. It is really interesting to see how different the performance is on the Internet workload!
On Thu, Sep 7, 2017 at 9:52 PM, dbohdan <danyil.boh...@gmail.com> wrote: > On Tuesday, September 5, 2017 at 12:41:04 PM UTC+3, Jay McCarthy wrote: >> Is the benchmarking client core the same core as the server core? >> Could that help explain why single threaded performance is best? > > The not-quite-yes-or-no answer is that they were limited to separate virtual > cores inside a VirtualBox VM. When a VirtualBox VM has N virtual cores on a > physical CPU with M cores, it can use roughly up to N/M of the CPU's > resources. In that benchmark the applications, which had access to two > virtual cores, had to themselves 50% of the four-core physical CPU, and the > load generator had 25% with one virtual core. > > I've run three variants of the benchmark to see if running in a VM had a > noticeable effect on single-threaded vs. multi-threaded performance and to > address your question about whether "many" underperformed because of a > virtual network. > > First, I got rid of the VM. Both the applications and the load generator ran > in containers on the same machine, but not in a VM. This meant they were > limited to different physical cores. The results were similar to those in a > VM. The numbers were lower overall due to the slightly weaker hardware. > > ====== > results/caddy.txt:Requests per second: 2758.06 [#/sec] (mean) > results/compojure.txt:Requests per second: 2670.11 [#/sec] (mean) > results/custom-many-places.txt:Requests per second: 4326.27 [#/sec] (mean) > results/custom-many.txt:Requests per second: 4655.04 [#/sec] (mean) > results/custom-places.txt:Requests per second: 4584.75 [#/sec] (mean) > results/custom-single.txt:Requests per second: 5191.93 [#/sec] (mean) > results/flask.txt:Requests per second: 1111.25 [#/sec] (mean) > results/guile.txt:Requests per second: 1933.10 [#/sec] (mean) > results/plug.txt:Requests per second: 3346.99 [#/sec] (mean) > results/scgi.txt:Requests per second: 2092.03 [#/sec] (mean) > results/sinatra.txt:Requests per second: 293.60 [#/sec] (mean) > results/stateful.txt:Requests per second: 532.61 [#/sec] (mean) > results/stateless.txt:Requests per second: 625.02 [#/sec] (mean) > ====== > > Second, I ran the benchmark over a gigabit local network. Yesterday I pushed > a script for this (`remote-benchmark.exp`) to the repository. The > applications ran on one machine (in a Docker container with access to two > virtual cores). The load generator ran on another (my laptop). > > ====== > remote-results/caddy.txt-Requests per second: 3119.23 [#/sec] (mean) > remote-results/compojure.txt-Requests per second: 4009.71 [#/sec] (mean) > remote-results/custom-many-places.txt-Requests per second: 4409.48 [#/sec] > (mean) > remote-results/custom-many.txt-Requests per second: 5499.20 [#/sec] (mean) > remote-results/custom-places.txt-Requests per second: 5072.63 [#/sec] > (mean) > remote-results/custom-single.txt-Requests per second: 6246.09 [#/sec] > (mean) > remote-results/flask.txt-Requests per second: 1106.43 [#/sec] (mean) > remote-results/guile.txt-Requests per second: 2062.53 [#/sec] (mean) > remote-results/plug.txt-Requests per second: 4034.74 [#/sec] (mean) > remote-results/scgi.txt-Requests per second: 2046.91 [#/sec] (mean) > remote-results/sinatra.txt-Requests per second: 288.52 [#/sec] (mean) > remote-results/stateful.txt-Requests per second: 542.27 [#/sec] (mean) > remote-results/stateless.txt-Requests per second: 614.18 [#/sec] (mean) > ====== > > In both cases the ordering is still "single" > "many" > "places" > > "many-places". Though "many" and "places" are pretty close in the first case, > "many" consistently comes out ahead if you retest. > > Last, I ran the benchmark over the Internet with two machines about > 1.89×10^-10 light years apart. The applications ran on a very humble VPS. Due > to its humbleness I had to reduce the number of concurrent connections to 25. > "places", "many-places", and racket-scgi ran out of memory with as few as 10 > concurrent connections (racket-scgi seemingly due to nginx), so I decided to > exclude them rather than reduce the number of connections further. > > ====== >> env CONNECTIONS=25 ./remote-benchmark.exp vps > remote-results/caddy.txt:Requests per second: 231.37 [#/sec] (mean) > remote-results/compojure.txt:Requests per second: 242.41 [#/sec] (mean) > remote-results/custom-many.txt:Requests per second: 250.35 [#/sec] (mean) > remote-results/custom-single.txt:Requests per second: 255.21 [#/sec] (mean) > remote-results/flask.txt:Requests per second: 235.26 [#/sec] (mean) > remote-results/guile.txt:Requests per second: 242.38 [#/sec] (mean) > remote-results/plug.txt:Requests per second: 244.98 [#/sec] (mean) > remote-results/sinatra.txt:Requests per second: 239.78 [#/sec] (mean) > remote-results/stateful.txt:Requests per second: 239.60 [#/sec] (mean) > remote-results/stateless.txt:Requests per second: 238.71 [#/sec] (mean) > ====== > > -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to racket-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- -=[ Jay McCarthy http://jeapostrophe.github.io ]=- -=[ Associate Professor PLT @ CS @ UMass Lowell ]=- -=[ Moses 1:33: And worlds without number have I created; ]=- -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
