Good points. I'm guess I'm really arguing that SPEC, TPC, SAP, Oracle Apps, Manugistics, SAS, etc are all good, real-world tests (and we should run them on OpenSolaris), but they're not Open tests. Not Open as in "Open Source", but Open as in "I have access to the tests, and I can run them on my hardware whenever I need/want to". They're a necessary part of good performance characterization, but IMHO, they're not sufficient.
Sun developers, of course, already have access to a large performance test infrastructure, that covers a set of machines (biased toward Sun's own hardware, no surprise there). OpenSolaris developers have to count on Sun (or some other company who can afford the $$) to run tests for them, and even then, it's only on the HW that Sun currently has setup. I think of it this way: it would be silly (OK, "non-optimal" :-) for OpenSolaris developers to have the Solaris source code, but not have the tools to build it (e.g. make, the compiler). Similarly, to not have access to the tools needed to run a full set of tests (including on HW that Sun does not own) limits the ability of non-Sun developers to write good code, and make good decisions. I'm very interested in encouraging non-Sun developers to join the effort, so it looks to me like there's a barrier here that needs removing. (By the way, I think this applies to more than just *performance* tests, too. It extends to *functional* tests as well.) Fortunately, at least part of this barrier is fixable. Two of the SPECcpu tests are Open, and it looks like they can be used as a proxy for the entire SPECcpu suite (with very little effort, I think). There are other benchmarks in common use that are Open as well (and, as a Sun manager working on performance, I do encounter customers who run direct Linux to S10 comparisons on their own, and <ehem> want us to explain why we are slower on some subtests). In other words, lots of people are running these other Open tests anyway. Having an Open Test Suite will enable more participation, I think, and I don't think the effort required is large. Oh, and yes, we'll be open sourcing libmicro (Sun's nifty micro-benchmark suite), too....so micro-benchmarks will not by any means be left out. :-) This message posted from opensolaris.org _______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
