Rajappa Iyer wrote:
>
> http://www.sysadminmag.com/articles/2001/0107/0107a/0107a.htm
>
> Any obvious reasons why FreeBSD performed so poorly for these people?
Here is a repeat of my post to -advocacy:
-- Terry
----
The article is meaningless.
Too bad they titled it "Which OS is Fastest for High-
Performance Network Applications?" instead of "Which OS is
Fastest for MailEngine?".
The only implied caveat is the statement "Our customers
frequently ask us which operating system is best for
running our software" in paragraph 3 of the "Background"
section. This should have been in bold type in the first
paragraph.
--
It makes a number of very large blunders, which are really
inexcusable, given that it tries to represent itself as a
fair and unbiased comparison.
These blunders are in the tuning of FreeBSD, the best
architecture for FreeBSD applications (one shich they did
not even try to consider), in their choice of which items
they could micro-benchmark would really be indicative of
real-world performance, and, finally, in their experimental
methodology.
Here is a short list, off the top of my head:
1) The mail server they were using doesn't come
with any of these systems "out of the box".
2) Threaded processes are vastly inferior to
finite state automatons, when it comes to CPU
utilization on single CPU systems, and even on
multiple CPU systems, if there is async I/O
that can be scheduled on multiple CPUs.
3) FreeBSD turns of write caching on IDE drives, by
default, in FreeBSD 4.3 and above; you can set
it to be like Linux, Solaris, and Windows, if
you don't care about your data. On FreeBSD 4.2
and below, Soft Updates are not enabled by
default. Either way, without tuning, you lose.
4) IDE drives do not support tagged command queueing,
except IBM DTLA drives, which are known to fail
due to overheating and due to their electronics
being too slow for their radial track density for
interior tracks.
5) Real servers with storage and I/O requirements
use SCSI drives so they can benefit from tagged
command queues, which allow I/O to be interleaved
instead of serialized.
6) No well designed mail server keeps all queue
files in the same directory, unless it has been
designed to run on a particular system where that
is not a problem; this is a design portability
issue, not a performance issue.
7) Sendmail can handle 400,000 8k emails in a 24 hour
period on a < 500MHz box, if it is properly set up
and queue dispersal is optimally configured (e.g.
with the patches from ftp://ftp.whistle.com/ ).
8) "The most efficient asynchornous architecture" for
an application is OS-dependent.
9) There are more than 3 ways to skin a cat, or to
architect a task.
10) Sending an RSET instead of data measures only the
connection setup and teardown speeds, and does not
measure real throughput, and is not representative
of real world behaviour, in which mail messages,
when sent, contain data, and not just trivial
addressing information.
11) Mail servers which support the ESMTP PIPELINE ability
have significantly higher throughput, even when just
doing addressing.
12) You can not "tweak" FreeBSD's network connection
limits at run time; socket structurse, inpcb's,
and tcpcb's are allocated via a zone allocator,
prior to the system actually being started. This
zone can not be resized. Without the patch I posted
to make maxfiles boot-time tunable, FreeBSD can not
increase the number of sockets and files that it can
simultaneously handle, without a kernel recompile.
Thus the "tweaking" used was useless.
13) For each connection, there is a tcptmpl structure,
which is used for keepalives. This structure will
consume one mbuf per connection; in addition, the
average TCP window size will be 16k; so you will
need 16k/2k (8) mbufs for custer pointers plus
16k/256 (64) mbufs for the window data, plus one
mbuf per connection, pplus one mbuf per connection,
if you are setting options. This means that you
will potentially need 74 mbufs per connection you
intend to support; without patching, this also is
not tunable except at compile time, and the value
was not tuned.
14) The "average througput per network architecture" is
extremely misleading, both because of the limited
and inefficient architectures used in the test, and
in using an average to "determine" which was "the
best architecture for use on all OSs". Per OS numbers
would have been much more meaningful, since the final
architecture was chosen based on the average, and not
based on what was best for the OS being tested.
15) Creation and deletion of large numbers of files in a
single directory is rather meaningless; witness the
moronic 'postmark' "benchmark", and the resulting
discussion on several FreeBSD lists. The "benchmark"
was clearly designed to put forth a political agenda
(in the context of the list discussion, this agenda
was the pro-ReiserFS position, despite Reiser's use
of several USL/Novell patents on Delayed Ordered
Writes, without the permission of the patent holders,
which render a pro-ResierFS argument as meaningless
as arguing about the number of angels which can dance
on the head of a pin).
16) When you have an answer you want, all the evidence you
gather will tend to support the conclusion you have
already drawn, unless you are very, very careful to
eliminate your assumed biases in your experimental
design. This is very hard to do, even when trained
scientists are involved.
Feel free to pass this on to the magazine editors, the author,
or both; at the very least, it may teach them a little bit
about how to actually tune FreeBSD to get good numbers out of
it, instead of frobbing meaningless knobs and merely _believing_
they are tuning the system for higher performance.
-- Terry
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
