I suggest you take a closer look Injong - there is a whole page of data
from tests covering a wide range of levels of background traffic. These
results are all new, and significantly strengthen the conclusions I
think, as is the expanded explanatory discussion of the observed
behaviour of the various algorithms (the result of a fair bit of
detective work of course). Your claim that "Your report mostly ignores
the effect of background traffic" is simply not true.
I can't really comment on your own tests without more information,
although I can say that we went to a good bit of trouble to make sure
our results were consistent and reproducible - in fact all our reported
results are from at least five, and usually more, runs of each test. We
were also careful to control for differences in kernel implementation so
that we compare congestion control algorithms rather than other aspects
of the network stack implementation. All of this is documented in the
paper. The kernel we used is available on the web. Our measurements
are also publicly available - the best way forward might be to pick one
or two tests and compare results of them in detail with a view to
diagnosing the source of any differences.
General comments such as "our experience tells that the RTT variations
of mid size flows play a very important role in creating significant
dynamics in testing environments" are not too helpful. What do you mean
by a "mid-sized flow" ? What do you mean by "significant dynamics" ?
What do you mean by "important role" - is this quantified ? Best to
stick to science rather than grandstanding. This is especially true
when dealing with a sensitive subject such as the evaluation of
competing algorithms.
Re FAST, we have of course discussed our results with the Caltech folks.
As stated in the paper, some of the observed behaviour seems to be
associated with the alpha tuning algorithm. Other behaviour seems to be
associated with packet burst effects that have also been reported
independently by the Caltech folks. Similar results to ours have since
been observed by other groups I believe. Perhaps differences between
our results point to some issue in your testbed setup.
Doug
Injong Rhee wrote:
This is a resend with fixed web links. The links were broken in my
previous email -- sorry about multiple transmissions.
---------------------------------------------------------------------------------
Hi Doug,
Thanks for sharing your paper. Also congratulations to the acceptance of
your journal paper to TONs. But I am wondering what's new in this paper.
At first glance, I did not find many new things that are different from
your previously publicized reports. How much is this different from the
ones you put out in this mail list a year or two ago and also the one
publicized in PFLDnet February this year
http://www.hpcc.jp/pfldnet2006/? In that same workshop, we also
presented our experimental results that shows significant discrepancy
from yours but i am not sure why you forgot to reference our
experimental work presented in that same PFLDnet. Here is a link to a
more detailed version of that report accepted to COMNET
http://netsrv.csc.ncsu.edu/highspeed/comnet-asteppaper.pdf
The main point of contention [that we talked about in that PFLDnet
workshop] is the presence of background traffic and the method to add
them. Your report mostly ignores the effect of background traffic. Some
texts in this paper state that you added some web traffic (10%), but the
paper shows only the results from NO background traffic scenarios. But
our results differ from yours in many aspects. Below are the links to
our results (the links to them have been available in our BIC web site
for a long time and also mentioned in our PFLDnet paper; this result is
with the patch that corrects HTCP bugs).
[Convergence and intra protocol fairness]
without background traffic:
http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/intra_protocol/intra_protocol.htm
with background traffic:
http://netsrv.csc.ncsu.edu/highspeed/1200/bk/intra_protocol/intra_protocol.htm
[RTT fairness]:
w/o background traffic:
http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/rtt_fairness/rtt_fairness.htm
with background traffic:
http://netsrv.csc.ncsu.edu/highspeed/1200/bk/rtt_fairness/rtt_fairness.htm
[TCP friendliness]
without background traffic:
http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/tcp_friendliness/tcp_friendliness.htm
with background traffic:
http://netsrv.csc.ncsu.edu/highspeed/1200/bk/tcp_friendliness/tcp_friendliness.htm
After our discussion in that PFLDnet, I puzzled why we get different
results. My guess is that the main difference between your experiment
and ours is the inclusion of mid-sized flows with various RTTs -- our
experience tells that the RTT variations of mid size flows play a very
important role in creating significant dynamics in testing environments.
The same point about the importance of mid size flows with RTT
variations has been raised in several occasions by Sally Floyd as well,
including in this year's E2E research group meeting. You can find some
reference to the importance of RTT variations in her paper too [
http://www.icir.org/models/hotnetsFinal.pdf]. Just having web-traffic
(all with the same RTTs) does not create a realistic environment as it
does not do anything about RTTs and also flow sizes tend to be highly
skewed with the Pareto distribution-- but I don't know exactly how you
create your testing environment with web-traffic -- I can only guess
from the description you have about the web traffic in your paper.
Another puzzle in this difference seems that even under no background
traffic, we also get different results from yours..hmm...especially with
FAST because under no background traffic, FAST seems to work fairly well
with good RTT fairness in our experiment. But your results show FAST has
huge RTT-unfairness. That is very strange. Is that because we have
different bandwidth and buffer sizes in the setup? I think we need to
compare our notes more. Also in the journal paper of FAST experimental
results [
http://netlab.caltech.edu/publications/FAST-ToN-final-060209-2007.pdf ],
FAST seems to work very well under no background traffic. We will verify
our results again in the exact same environment as you have in your
report, to make sure we can reproduce your results....but here are some
samples of our results for FAST.
http://netsrv.csc.ncsu.edu/highspeed/1200/nobk/rtt_fairness/1200--2.4_FAST-2.4_FAST-NONE--400-3-1333--1000-76-3-0-0-0-5-500--200000-0.6-1000-10-1200-64000-150--1/
In this experiment, FAST flows are just perfect. Also the same result is
confirmed inthe FAST journal paper [
http://netlab.caltech.edu/publications/FAST-ToN-final-060209-2007.pdf --
please look at Section IV.B and C. But your results show really bad RTT
fairness.]
Best regards,
Injong
---
Injong Rhee
NCSU
On Sep 22, 2006, at 10:22 AM, Douglas Leith wrote:
For those interested in TCP for high-speed environments, and perhaps
also people interested in TCP evaluation generally, I'd like to point
you towards the results of a detailed experimental study which are now
available at:
http://www.hamilton.ie/net/eval/ToNfinal.pdf
<http://www.hamilton.ie/net/eval/ToNfinal.pdf>
This study consistently compares Scalable-TCP, HS-TCP, BIC-TCP, FAST-TCP
and H-TCP performance under a wide range of conditions including with
mixes of long and short-lived flows. This study has now been subject to
peer review (to hopefully give it some legitimacy) and is due to appear
in the Transactions on Networking.
The conclusions (see summary below) seem especially topical as BIC-TCP
is currently widely deployed as the default algorithm in Linux.
Comments appreciated. Our measurements are publicly available - on the
web or drop me a line if you'd like a copy.
Summary:
In this paper we present experimental results evaluating the
performance of the Scalable-TCP, HS-TCP, BIC-TCP, FAST-TCP and
H-TCP proposals in a series of benchmark tests.
We find that many recent proposals perform surprisingly poorly in
even the most simple test, namely achieving fairness between two
competing flows in a dumbbell topology with the same round-trip
times and shared bottleneck link. Specifically, both Scalable-TCP
and FAST TCP exhibit very substantial unfairness in this test.
We also find that Scalable-TCP, HS-TCP and BIC-TCP induce significantly
greater RTT unfairness between competing flows with different round-trip
times. The unfairness can be an order of magnitude greater than that
with standard TCP and is such that flows with longer round-trip times
can be completely starved of bandwidth.
While the TCP proposals studied are all successful at improving
the link utilisation in a relatively static environment with
long-lived flows, in our tests many of the proposals exhibit poor
responsiveness to changing network conditions. We observe that
Scalable-TCP, HS-TCP and BIC-TCP can all suffer from extremely
slow (>100s) convergence times following the startup of a new
flow. We also observe that while FAST-TCP flows typically converge
quickly initially, flows may later diverge again to create
significant and sustained unfairness.
--Doug
Hamilton Institute
www.hamilton.ie <http://www.hamilton.ie/>
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
