On Thu, May 13, 2010 at 8:13 PM, wren ng thornton <[email protected]> wrote: > Andrea Vezzosi wrote: >> >> On Thu, May 13, 2010 at 10:51 AM, wren ng thornton <[email protected]> >> wrote: >>> >>> Andrea Vezzosi wrote: >>>> >>>> wren ng thornton wrote: >>>>> >>>>> With this change [1] I can't notice any difference for your >>>>> benchmark[2]. >>>>> Then again, all the runTest calls take 0 msec and I've had no luck >>>>> making >>>>> the computation take much time; perhaps your computer can detect a >>>>> difference. >>>> >>>> On my machine, with ghc-6.12.1, yours and the original ErrCPS give >>>> quite similar results, both ~2x slower than Either. >>>> However it's important to note that these results are highly dependent >>>> on the monadic expressions being evaluated, with a different benchmark >>>> you can get an huge speedup with the CPS versions. >>> >>> That's very curious. After installing Criterion, my machine (OSX 10.5.8 >>> 2.8GHz Intel Core2Duo, GHC 6.12.1 with -O2) shows only 1% difference >>> between >>> my ErrCPS and Either on this benchmark. Alas, I can't print kernel >>> density >>> graphs since Crieterion charts are broken on 6.12. It seems buggy that >>> your >>> platform would behave so much differently... >> >> I got the measurements from the original code, could you share the >> code that uses criterion instead? > > The 1% number was buggy because I hadn't factored the generation of random > lists out of the benchmark. But, having fixed that, I still can't replicate > your numbers: I get 12us for Either, vs 17us for EitherCPS. > > http://community.haskell.org/~wren/wren-extras/test/Control/Monad/ErrCPS/CriterionBenchmark.hs > > > > Yet another version of the same benchmark, this time using Microbench: > > http://community.haskell.org/~wren/wren-extras/test/Control/Monad/ErrCPS/MicrobenchBenchmark.hs > > Microbench seems to replicate your numbers better: 2551.930ns vs 4466.832ns > (or 391.86 vs 223.87 calls per second)--- though this is getting into the > range where there might be Int overflow issues corrupting the results (a > similar problem showed up when benchmarking Data.Trie vs Data.Map), so it > may warrant further investigation. >
That might be the case, i'm on 64bit: sai...@astarte:~$ uname -a Linux astarte 2.6.32-ARCH #1 SMP PREEMPT Tue Feb 23 19:43:46 CET 2010 x86_64 Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz GenuineIntel GNU/Linux sai...@astarte:~$ ./CriterionBenchmark warming up estimating clock resolution... mean is 6.834442 us (80001 iterations) found 1240 outliers among 79998 samples (1.6%) 1131 (1.4%) high severe estimating cost of a clock call... mean is 107.2316 ns (41 iterations) benchmarking Either collecting 100 samples, 1039 iterations each, in estimated 683.8220 ms bootstrapping with 100000 resamples mean: 6.563462 us, lb 6.553649 us, ub 6.570454 us, ci 0.950 std dev: 41.74602 ns, lb 23.76971 ns, ub 67.67842 ns, ci 0.950 found 8 outliers among 100 samples (8.0%) 2 (2.0%) low severe 4 (4.0%) high mild 2 (2.0%) high severe variance introduced by outliers: 0.990% variance is unaffected by outliers benchmarking ErrCPS collecting 100 samples, 1 iterations each, in estimated 1.334000 s bootstrapping with 100000 resamples mean: 13.14468 ms, lb 13.10442 ms, ub 13.18208 ms, ci 0.950 std dev: 198.3150 us, lb 182.0600 us, ub 220.7957 us, ci 0.950 variance introduced by outliers: 0.993% variance is unaffected by outliers If i'm reading it correctly this gives even worse results: 6us vs. 13ms > -- > Live well, > ~wren > _______________________________________________ > Haskell-Cafe mailing list > [email protected] > http://www.haskell.org/mailman/listinfo/haskell-cafe > _______________________________________________ Haskell-Cafe mailing list [email protected] http://www.haskell.org/mailman/listinfo/haskell-cafe
