Not a problem.I apologize for the critical tone (but also perhaps ignorance)of my [now next to the] last post. I've got the flu or something. Yecch.
That's right ... once a servlet is loaded it stays in memory, so there would be little or no I/O in this kind of test scenario.Here's the data I collected after running new, slightly modified, scripts (see attached) for which RequestInfoExample is requested by ab 100,000 times on a Celeron 400Mz, 256MB SD100 memory, WD Caviar disk, Redhat6.1, with almost nothing else going on:
1 2912.78 34.33 0 28 100 0.87 52.2 2860.58 28.6 10 1495.12 66.88 0 149 100 0.92 55.2 14.4 20 1488.7 67.17 0 297 100 0.98 58.8 1429.9 14.3 30 1503.03 66.53 0 450 100 1.12 67.2 1435.83 14.4 40 1591.7 62.83 0 636 100 1.31 78.6 1513.1 15.1 The earlier results involving the HelloWorldExample servlet were affected by variability which I think/hope is averaged out by the large number of requests made. The disk, as expected shows only about 1.8 requests per second, and is not the bottleneck anyway (as expected).
NOTE: Of course, that is one of the issues that leads to caution in interpreting benchmark results as *absolute* indicators of performance -- this is clearly not going to be representative of the performance of a servlet that heavily accesses a database, for example. However, it is very useful in *relative* performance measurement. The standard practices of scientific investigation apply ... tweak one variable at a time and see what happens.
My earlier results concerning HelloWorldExample were based on a smaller N and were not adjusted to account for CPU overhead. I'll re-run them ASAP. In the earlier results, thread queue locking was observed. Craig noted that the number of background threads is set to 75 in the config file. With that in mind, I made sure that the ab script ran with 70 or fewer concurrent requests. Here's what happened: catalina_log*.txt shows "HttpProcessor starting background thread[8080][i]" where i runs from 0/1 up to 50. catalina.out contains a full thread dump (50 on down), one of which involves the message "Waiting to be notified HttpProcessor[8080][43]". This appears to be the result of a segmentation violation involving HttpProcessor[8080][50] that is also reported. Of course, ab barfs from C =3D 50 onward. As I recall, SEGV was also the cause of the thread dump=
In a pure Java program like Tomcat 4.0-m4, any SEGV violation is always, 100% of the time, a bug in the JVM (or the underlying OS -- and Linux thread libraries have been known to have issues). I don't recall which JVM you are running, but you might want to try at least the 1.2.2 and 1.3.0 releases from Sun (on the same hardware) to compare. That would also give some interesting information on how the new HotSpot in 1.3 (and in particular the "server version") affects things.when I ran HelloWorldExample. So, the thread dump seems to be a repeatable phenomena (on my machine), apparently a function of N & C. QUESTION 1: Does segmentation violation occur on other machines? If so, is it a problem? If not, why not?
As discussed above, I would be cautious about treating these numbers as absolute indicators of expected performance. What we are after is identifying places where we can change the code and improve the simple cases, with the expectation that such improvements will also help in more realistic cases (even though they will have less impact on the overall performance, because disk times for database access will tend to dominate).QUESTION 2: What do the thruput and average processing times mean? Are they "good", "bad", or what?
When designing a server platform, most people aim for goals like this:
* Maximum processing times should not have huge
spikes versus average
* Graceful degredation (ideally linear) as the load
is increased
Deviations from these goals usually indicate a bottleneck that is being bumped into, and provides opportunities for tuning.
One thing that will help in this regard is that Sun is making a machine with some serious hardware (six processors, 3GB of memory, scads of disk) available for performance tuning. That should help us shake out thread-related bugs and performance problems pretty quickly :-).
I agree with this, and would word it slightly differently: the only thing a "Hello, World" benchmark predicts is how fast your application can say "Hello, World". :-)QUESTION 3: Since the scripts can be modified to run any servlet, how about a "realistic" one as Costin suggested in his ApacheCon presentation (unless the examples are "representative").
If you're after a performance predictor for a real application, you need a test suite that simulates the actual behavior of the system as closely as possible -- particularly where the stresses are bumping up against bottlenecks. For example, in the test suite you've quoted the system runs at 100% CPU, so switching to a faster processor (all things else being equal) would probably make everything run faster -- but adding a faster disk (or even more memory, if you're not swapping) would not matter at all.
Benchmarks for performance prediction are an interesting art.
(Hack) Roy Roy Wilson E-mail: [EMAIL PROTECTED]
# Draft script by Roy Wilson used on Redhat6.1 system (bash) # Assume ONLY HelloWorld servlet being used: better to have a set, but what? # # To promote comparability, run script immediately after a cold boot echo Run script only after a cold boot to make results comparable echo When done please and post the summary file to Tomcat dev list echo Usage: echo sh tcscript servlet-name max-concurrency increment echo Example: echo sh tcscript HelloWorldExample 100 10 rm -f summary sh tcscript 1 $1 load="$3" while [ "$load" -le "$2" ] do echo "Starting ab with load = " $load sh tcscript $load $1 load=$(($load+$3)) done
# Roy Wilson 11/2/00 echo "Starting ab load test with C = " $1 "and servlet = "$2 rm -f $1$2.sardata # Next 2 lines may need adjustment on account of OS differences # Start with a fresh temporary system accounting log file date >> summary cp /var/log/savacct /var/log/tcacct /usr/sbin/accton /var/log/tcacct # sample CPU and disk usage with 1 second resolution (sar -u -b -o $1$2.sardata 1 > /dev/null )& sarpid=$! # modify path in next line for use of servlet /usr/sbin/ab -n 100000 -c $1 http://ALFRED:8080/examples/servlet/$2 >> summary kill -s TERM ${sarpid} # Next line may need modification: Turns off system accounting /usr/sbin/accton # peel off raw data, keep sar summary line sar -f $1$2.sardata 1 | grep Average >> summary # collect CPU per minute data from system accounting package /usr/sbin/sa -a /var/log/tcacct >> summary rm -f $1$2.sardata echo "Finished ab load test with C = "$1 "and servlet = "$2
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
