Hi, I've repeated the tests on the hybrid architecture using the AB. You can find them attached to this mail. I've run the AB with several concurrency levels, ranging from 20 to 10000. You can see all the results in a plot.
> Running a test with ab (ab -k -c 20 -n 20000 > http://host:8080/tomcat.gif) would take 30s, and would make comparisons > easy (basically, I actually know what the tests do ...), and will be an > actual measurement of throughput. I've been studying the behavior of the AB and I've several doubts about the significance of the results when trying to measure the throughtput of a server. In my opinion, the AB is a great way to test the impact of architectural modifications on the internal latency of the tomcat execution pipeline, but not a deterministic way to compare the throughput of two servers. In the following paragraphs I try to justify this idea (hope that in a comprensible way) :) The first thing that makes me suspect about the reliability of the obtained results is that this benchmark produces a different workload intensity for each tested server. I mean that, given a number of concurrent clients simulated, the AB produces a higher number of requests as lower is the response time for the server (a new request is issued when the previous is completed). This behavior will always favour an architecture with lower internal latencies, even when it manages concurrency worse. This is the case of the tomcat multithreaded architecture. Other architectures, for instance the Hybrid or any other using non-blocking operations with readiness selectors, will always obtain worse results for low loads (remember that the select operation introduces an internal latency of 15-30ms since data is ready in a channel, with the purpose of getting more channels ready during that period). When the simulated number of concurrent clients is increased (and especially when the number of threads in the pool is lower than the number of emulated clients), the multithreaded architecture starts suffering. You can check the plots for the throughput, number of keep-alive requests, errors or connect time to create your own opinion. In conclusion, it must be taken into account that using this benchmark to compare the throughput of several architectural proposals can lead to wrong conclusions, especially when WANs (instead of fast LANs) are used for the evaluation. Let me explain with a simple example, assuming that the AB simulated 10 concurrent clients... Lets suppose two architectures, A and B: A -> Internal service time for the tomcat.gif file: 1ms B -> Internal service time for the tomcat.gif file: 5ms And lets suppose two different scenarios: First, assuming a zero network latency... (in practice, a Gbit LAN) - The observed throughput for A will be 10000 replies/sec (10*(1/10^-3)) - The observed throughput for B will be 2000 replies/sec (10*(5/10^-3)) - The speedup observed between A and B is 5. Later, assuming a 200 ms network latency... (in practice, a WAN) - The observed throughput for A will be 49,75 replies/sec (10*(1/(10^-3 + 200^-3))) - The observed throughput for B will be 48,78 replies/sec (10*(1/(5^-3 + 200^-3))) - The speedup observed between A and B is 1,019. This reasoning indicates that this test is more precise to compare the response time between two server architectures than to evaluate its performance, because the network latency (between the server and the client) can bias the obtained results. Finally, I miss a "think-time" as one of the configuration parameters of the AB. It reduces the "realism" of the test and makes not possible to test the performance of the server in terms of "user sessions" instead of individual requests. > > Note: your patch still seems bad, as the file add is represented as a > diff. This is likely not going to be patcheable. I'm very sorry but I missunderstood you again... :( Again... do I send the modified files "as is", instead of a diff? Thanks for the comments, Vicenç PS: I'm very interested in your opinion (I mean the community) about my reasoning about the adequate use of the AB for throughput comparisons... > > Rémy > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
