On Wed, 2008-02-13 at 14:21 +0000, [EMAIL PROTECTED] wrote: > > > > On a more rational note, has any thought been given to what "good enough > > performance for release" will be? > > Should we perhaps add a performance benchmark to the tests? > > Normalising it to account for hardware variations might be a problem.
There are existing benchmarks in examples/benchmarks/ , which have mostly solved the problem of hardware variations [1] by comparing instead against the performance of other dynamic language implementations. Certainly different versions of those other language implementations will have varying performance, but we can standardize our comparison on, say, the most recent release of each. There is also the language shootout, which has quite a few more comparisons available. My personal feeling is that for each comparison benchmark, we should be comparing PIR and Rakudo performance against the fastest other language implementation for that benchmark. Then we should be looking for two problems -- relatively consistent poor performance (indicating a general implementation design problem), and particular benchmarks that do unusually poorly (indicating some particular axis of pain). Some of these problems will be solveable, others will be amenable to workarounds, and others will probably just need to be documented. But we should really make sure we know as much as possible what the problems *are* before the first production release. -'f [1] Mind you, we still need to know if for example PPC handles method calls way slower than Intel, so we can determine if there is something we can do about that. But that is another story.
