>From the benchstat help text:
benchstat -h
usage: benchstat [options] old.txt [new.txt] [more.txt ...]
options:
-alpha α
consider change significant if p < α (default 0.05)
....
Specifically the "p" in your results is reported to be at least 0.100
which is probably because you have a small sample size with N=3. Try
again with N=10 and see if that helps.
On Tue, Jul 27, 2021 at 9:42 PM Kurtis Rader <kra...@skepticism.us> wrote:
>
> You didn't show us the actual commands you are running to produce the two log
> files. It is also unclear why you expect benchstat to report a difference. In
> other words: insufficient information for anyone to make even a guess why
> benchstat isn't producing the output you expect.
>
> On Tue, Jul 27, 2021 at 8:56 PM Tong Sun <suntong...@gmail.com> wrote:
>>
>> Hi,
>>
>> I'm trying to understand the benchstat's output.
>> The benchstat that I have is fresh from golang.org/x/perf/cmd/, installed
>> hours ago.
>>
>> This is understandable:
>>
>> name old time/op new time/op delta
>> FormatEmoji-2 1.79µs ± 0% 1.82µs ± 1% ~ (p=0.100 n=3+3)
>> FormatEmoji-4 1.99µs ± 1% 1.91µs ± 4% ~ (p=0.200 n=3+3)
>> FormatEmoji-8 1.97µs ± 3% 1.99µs ± 1% ~ (p=0.700 n=3+3)
>> FormatEmoji-12 2.05µs ± 1% 2.03µs ± 1% ~ (p=0.400 n=3+3)
>>
>> name old alloc/op new alloc/op delta
>> FormatEmoji-2 384B ± 0% 384B ± 0% ~ (p=1.000 n=3+3)
>> FormatEmoji-4 386B ± 0% 385B ± 0% ~ (p=0.700 n=3+3)
>> FormatEmoji-8 387B ± 0% 386B ± 0% ~ (p=0.400 n=3+3)
>> FormatEmoji-12 388B ± 0% 388B ± 0% ~ (p=1.000 n=3+3)
>>
>> name old allocs/op new allocs/op delta
>> FormatEmoji-2 7.00 ± 0% 7.00 ± 0% ~ (all equal)
>> FormatEmoji-4 7.00 ± 0% 7.00 ± 0% ~ (all equal)
>> FormatEmoji-8 7.00 ± 0% 7.00 ± 0% ~ (all equal)
>> FormatEmoji-12 7.00 ± 0% 7.00 ± 0% ~ (all equal)
>>
>> However,
>>
>> Why benchstat thinks there is no differences between the following two runs?
>>
>> $ benchstat /tmp/old.log /tmp/new.log
>> name old time/op new time/op delta
>> FormatEmojiString-2 1.79µs ± 0% 6.81µs ± 2% ~ (p=0.100 n=3+3)
>> FormatEmojiString-4 1.99µs ± 1% 7.22µs ± 1% ~ (p=0.100 n=3+3)
>> FormatEmojiString-8 1.97µs ± 3% 7.55µs ± 2% ~ (p=0.100 n=3+3)
>> FormatEmojiString-12 2.05µs ± 1% 7.64µs ± 1% ~ (p=0.100 n=3+3)
>>
>> name old alloc/op new alloc/op delta
>> FormatEmojiString-2 384B ± 0% 1459B ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiString-4 386B ± 0% 1463B ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiString-8 387B ± 0% 1468B ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiString-12 388B ± 0% 1473B ± 0% ~ (p=0.100 n=3+3)
>>
>> name old allocs/op new allocs/op delta
>> FormatEmojiString-2 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiString-4 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiString-8 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiString-12 7.00 ± 0% 25.00 ± 0% ~ (p=0.100 n=3+3)
>>
>> I'd think that the delta will be quite big.
>>
>> Now take a look at this. benchstat thinks there is no differences between
>> the following two runs is beyond imagination:
>>
>> $ benchstat /tmp/old.log /tmp/new.log
>> name old time/op new time/op delta
>> FormatEmojiBlock-2 1.79µs ± 0% 13543.09µs ± 4% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-4 1.99µs ± 1% 14597.09µs ± 1% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-8 1.97µs ± 3% 16390.62µs ± 2% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-12 2.05µs ± 1% 17409.37µs ± 2% ~ (p=0.100 n=3+3)
>>
>> name old alloc/op new alloc/op delta
>> FormatEmojiBlock-2 384B ± 0% 15270330B ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-4 386B ± 0% 15270671B ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-8 387B ± 0% 15271206B ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-12 388B ± 0% 15271727B ± 0% ~ (p=0.100 n=3+3)
>>
>> name old allocs/op new allocs/op delta
>> FormatEmojiBlock-2 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-4 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-8 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3)
>> FormatEmojiBlock-12 7.00 ± 0% 3467.00 ± 0% ~ (p=0.100 n=3+3)
>>
>> What's happening? What I've done wrong?
>>
>> Thx
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/a56a17b1-a8c3-4926-a27a-21cd86074f67n%40googlegroups.com.
>
>
>
> --
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD8R2NBveVArAchBOkrL1qZ6oNRpFqWLqbF7aKG5o3L%2Bxw%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAE1MmuFSVOFqbvtj0xNC11HFZT_TebMm-RCuXHHWZjtVAk4RHQ%40mail.gmail.com.
