>I try to follow the Tolly Group who compares products, and they
>continually show that Cisco equipment
>is a poor performer in almost any equipment compared to others, I find
>that so hard to believe.....
Just a rough comment here. Tolly's business model is a sponsored test
one, and Cisco is the dominant vendor in the marketplace. So when
anyone has a new product they want to promote, they can hire Tolly's lab
to show how a particular feature set is implemented and can often show
how one model exceeds Cisco's performance on some or all benchmarks.
While Tolly never cheats, remember that the guys who "win" the test are
the ones that are helping to design the test methodology, and so they
are focusing on features that they know that they can beat Cisco on.
Thus, the Tolly reports are true, but may not be telling the whole story
or may be focusing on a particular feature more than anything else. No
one is lying to you, but you may not be given the whole picture.
There's nothing wrong with having Tolly show that your product is in the
same league as Cisco's, but I would not necessarily read those reports
to say that Cisco is always the "loser" in any new product comparison.
The real value in Tolly's reports is to give credibility to the new and
up-coming products by having a mostly-independent source show that the
new products are comparable with the market leaders' in some ways.
Tolly is also very very important at doing spec verification: showing
that a product actually DOES what the specification sheet says it does.
Remember that those numbers are often made up by product managers
based on poorly-done in-house tests, so having a third-party, even a
paid one, say "yes, it does do that in our test lab" is WAY more
information than you get from most vendors.
Tolly provides a valuable service to our community but you need to look
at a combination of benchmarks from Tolly as well as other very-unbiased
sources (IDG and CMP magazines are pretty thorough) AND YOUR OWN TESTING
to know what is best for you.
On the other hand, compared to the analyst firms like Gartner who don't
even have test labs and generate their reports based on phone calls,
Google searches, and unverified specification sheets, Tolly is a source
of infinite knowledge about products.
Finally, also remember that performance is only one part of system
design and deployment. Management and feature completeness are, for the
operator community, often more important than whether the box can do 80
Gbps or 85 Gbps. Product selection is best done by YOU setting YOUR
requirements and doing YOUR OWN testing, not by looking at point tests
to see if one product version/model/device beats another in a particular
small set of performance tests.
Obligatory disclaimer: we don't compete directly with Tolly, but we do
get paid by the magazines to do testing and there is some overlap, so we
do compete indirectly.
jms
--
Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
Senior Partner, Opus One Phone: +1 520 324 0494
j...@opus1.com http://www.opus1.com/jms
