Re: Polygraph Considered Evil 8^) (was: Re: 5-STABLE Roadmap)

2003-02-17 Thread Alex Rousskov
On Mon, 17 Feb 2003, Terry Lambert wrote: > First, I just have a slight editorial comment, about cheating on > Polygraph. Terry, This is not the place to start a long discussion about our Polygraph testing methodology, but I have to say, with all due respect, that many of your statements

Re: Polygraph Considered Evil 8^) (was: Re: 5-STABLE Roadmap)

2003-02-17 Thread Terry Lambert
Jonathan Lemon wrote: > In article [EMAIL PROTECTED]> you write: > >Alex Rousskov wrote: > >One issue I have with Polygraph is that it intentionally works > >for a very long time to get worst case performance out of caches; > >basically, it cache-busts on purpose. Then the test runs. This > >seem

Re: Polygraph Considered Evil 8^) (was: Re: 5-STABLE Roadmap)

2003-02-17 Thread Jonathan Lemon
In article [EMAIL PROTECTED]> you write: >Alex Rousskov wrote: >One issue I have with Polygraph is that it intentionally works >for a very long time to get worst case performance out of caches; >basically, it cache-busts on purpose. Then the test runs. This >seems to be an editorial comment on en

Polygraph Considered Evil 8^) (was: Re: 5-STABLE Roadmap)

2003-02-17 Thread Terry Lambert
Alex Rousskov wrote: > Polygraph is relatively easy to setup on FreeBSD for standard tests, > using two PCs. Testing with more PCs, with non-standard workloads, > and/or on a regular basis requires writing scripts and can get pretty > evolved (which let's us sell a pre-configured appliance that doe

Re: 5-STABLE Roadmap

2003-02-17 Thread Hiten Pandya
On Thu, Feb 13, 2003 at 04:36:20PM -0800, Scott Long wrote the words in effect of: > - Benchmarks and performance testing - Having a source of reliable and >useful benchmarks is essential to identifying performance problems >and guarding against performance regressions. A 'performance tea

Re: 5-STABLE Roadmap

2003-02-16 Thread Alex Rousskov
On Sun, 16 Feb 2003, Pawel Jakub Dawidek wrote: > On Sun, Feb 16, 2003 at 02:08:35PM -0700, Scott Long wrote: > +> Pawel Jakub Dawidek wrote: > +> > +> >On Thu, Feb 13, 2003 at 08:28:43PM -0800, Sam Leffler wrote: > +> >+> This can quickly turn into a bikeshed, but suggest ones. We're > +> >looki

Re: 5-STABLE Roadmap

2003-02-16 Thread Pawel Jakub Dawidek
On Sun, Feb 16, 2003 at 02:08:35PM -0700, Scott Long wrote: +> Pawel Jakub Dawidek wrote: +> +> >On Thu, Feb 13, 2003 at 08:28:43PM -0800, Sam Leffler wrote: +> >+> This can quickly turn into a bikeshed, but suggest ones. We're +> >looking for +> >+> good benchmarks. [...] +> > +> >Look at: +> >

Re: 5-STABLE Roadmap

2003-02-16 Thread Scott Long
Pawel Jakub Dawidek wrote: On Thu, Feb 13, 2003 at 08:28:43PM -0800, Sam Leffler wrote: +> This can quickly turn into a bikeshed, but suggest ones. We're looking for +> good benchmarks. [...] Look at: http://www.web-polygraph.org It provides tests for www-cache/proxy stuff. We can test many

Re: 5-STABLE Roadmap

2003-02-16 Thread Pawel Jakub Dawidek
On Thu, Feb 13, 2003 at 08:28:43PM -0800, Sam Leffler wrote: +> This can quickly turn into a bikeshed, but suggest ones. We're looking for +> good benchmarks. [...] Look at: http://www.web-polygraph.org It provides tests for www-cache/proxy stuff. We can test many things with it:

Re: 5-STABLE Roadmap

2003-02-14 Thread Adam Migus
> - Benchmarks and performance testing - Having a source of > reliable and >useful benchmarks is essential to identifying performance > problems and guarding against performance regressions. A > 'performance team' that is made up of people and resources > for formulating, developing, and ex

Re: 5-STABLE Roadmap

2003-02-14 Thread Darrell Anderson
Hey Drew, thanks for the heads up. I'd be happy to see Fstress included with FreeBSD! Regarding SPECsfs, Fstress has a mode that emulates it exactly. A number of major storage companies have adopted it as an easier way to tune their systems before running the actual SPECsfs release numbers. (

Re: 5-STABLE Roadmap

2003-02-14 Thread Brad Knowles
At 9:47 PM -0800 2003/02/13, Sam Leffler wrote: SpecFS (NFS ops/sec benchmark) List price on SPEC SFS97 R1 is $900. And my recollection is that it was involved to setup and run. $450 for educational organizations. Wouldn't the FreeBSD Foundation qualify? Benchmarks must be unencu

Re: 5-STABLE Roadmap

2003-02-14 Thread Brad Knowles
At 8:28 PM -0800 2003/02/13, Sam Leffler wrote: This can quickly turn into a bikeshed, but suggest ones. We're looking for good benchmarks. lmbench, rawio, and bonniee are rather "micro" in nature (not bad, just limited in their usefulness). Well, I would submit that webstone and ApacheBe

Re: 5-STABLE Roadmap

2003-02-14 Thread Brad Knowles
At 6:53 AM -0800 2003/02/14, Sam Leffler wrote: $450 for educational organizations. Wouldn't the FreeBSD Foundation qualify? The point was that they cost $$$. Not an option for many developers. Fair enough. Microbenchmarks are valuable here and hav

Re: 5-STABLE Roadmap

2003-02-14 Thread Sam Leffler
> At 9:47 PM -0800 2003/02/13, Sam Leffler wrote: > > >> SpecFS (NFS ops/sec benchmark) > > > > List price on SPEC SFS97 R1 is $900. And my recollection is that it was > > involved to setup and run. > > $450 for educational organizations. Wouldn't the FreeBSD > Foundation qualify? > The p

Re: 5-STABLE Roadmap

2003-02-14 Thread Andrew Gallatin
Chris BeHanna writes: > > > At 4:36 PM -0800 2003/02/13, Scott Long wrote: > > > > - Fstress - http://www.cs.duke.edu/ari/fstress > > SpecFS (NFS ops/sec benchmark) > Have you ever actually used SPECsfs97? In addition to being encumbered, SPECsfs97 is pain to keep running (dies a

Re: 5-STABLE Roadmap

2003-02-13 Thread Sam Leffler
> On Thursday 13 February 2003 11:28 pm, Sam Leffler wrote: > > > At 4:36 PM -0800 2003/02/13, Scott Long wrote: > > > > - the classic 'worldstone' > > > > - webstone - /usr/ports/www/webstone > > > > - Fstress - http://www.cs.duke.edu/ari/fstress > > > > - ApacheBench - /usr/po

Re: 5-STABLE Roadmap

2003-02-13 Thread Chris BeHanna
On Thursday 13 February 2003 11:28 pm, Sam Leffler wrote: > > At 4:36 PM -0800 2003/02/13, Scott Long wrote: > > > - the classic 'worldstone' > > > - webstone - /usr/ports/www/webstone > > > - Fstress - http://www.cs.duke.edu/ari/fstress > > > - ApacheBench - /usr/ports/www/p5-A

Re: 5-STABLE Roadmap

2003-02-13 Thread Sam Leffler
> At 4:36 PM -0800 2003/02/13, Scott Long wrote: > > > - the classic 'worldstone' > > - webstone - /usr/ports/www/webstone > > - Fstress - http://www.cs.duke.edu/ari/fstress > > - ApacheBench - /usr/ports/www/p5-ApacheBench > > - netperf - /usr/ports/benchmarks/netperf > >

Re: 5-STABLE Roadmap

2003-02-13 Thread Sam Leffler
> About benchmarks... > > FWIW, the reiserfs people were excited about SCO's > release of AIM: > > http://caldera.com/developers/community/contrib/aim.html > > but the announcement went rather unnoticed in > freebsd-fs. > Thanks. I've worked with AIM. Wasn't aware it had been released.

Re: 5-STABLE Roadmap

2003-02-13 Thread Pedro F. Giffuni
About benchmarks... FWIW, the reiserfs people were excited about SCO's release of AIM: http://caldera.com/developers/community/contrib/aim.html but the announcement went rather unnoticed in freebsd-fs. cheers, Pedro. __ Y

Re: 5-STABLE Roadmap

2003-02-13 Thread Brad Knowles
At 4:36 PM -0800 2003/02/13, Scott Long wrote: - the classic 'worldstone' - webstone - /usr/ports/www/webstone - Fstress - http://www.cs.duke.edu/ari/fstress - ApacheBench - /usr/ports/www/p5-ApacheBench - netperf - /usr/ports/benchmarks/netperf Are there any other ben

5-STABLE Roadmap

2003-02-13 Thread Scott Long
All, Thanks to the hard work of everyone, FreeBSD 5.0 became a reality and is working better than most even hoped. However, there is still a lot of work to be done before we can create the RELENG_5/5-STABLE branch and declare success. Below is a document that I have drafted with the input and re