On Tuesday, 24 March 2020 16:49:55 UTC, Robert Engels wrote:
>
> Can you please succinctly explain the problem?
>
Let's see. Benchmarks can yield incorrect results when b.StopTimer and
b.StartTimer are used.
My reasoning:
1/ I have a benchmark that calls a single function. The reported duration
is T.
2/ I have another benchmark which invokes the same function twice. I'd
expect the reported duration for this to be nominally 2T and within reason
it appears to be the case.
3/ I have yet another benchmark which also invokes the same function twice
but it stops the benchmark timer before the first call and then starts it
after the the first call has completed. I'd expect the reported duration to
be nominally T again. It isn't. It's closer to 3T on my hardware and I've
seen much worse correlation on other hardware - almost 5T
Now it's entirely possible that I'm misunderstanding something about how
b.StopTimer and b.StartTimer are intended to be used, hence my post here :)
>
>
> On Mar 24, 2020, at 11:24 AM, Orson Cart <objectiv...@gmail.com
> <javascript:>> wrote:
>
>
> I posted this earlier but I realised that the code had a fundamental error
> in it. I've corrected here it but the underlying problem still exists.
>
> I've recently started using go test's benchmarks support and I'm
> particularly interested in understanding the benchmark timer functions.
> I've been getting results that I found surprising and I was wondering if
> anyone could explain what's going on here.
>
> The code below has three benchmarks that each invoke a single function
> (foo). The implementation of foo isn't important, it's just there to
> consume some time:
> - foo is called once per iteration in Benchmark1.
> - It's called twice per iteration in Benchmark2 so I'd expect Benchmark2's
> duration to be nominally twice that of Benchmark1.
> - It's also called twice per iteration in Benchmark3 but the first call is
> wrapped in b.StopTimer and b.startTimer calls. Because of this I'd have
> expected Benchmark3 to be about the same duration as Benchmark1
>
> Apologies for the length of the example but I didn't think it fair to ask
> the question and leave anything out.
>
> package demo_test
>
> import (
> "strconv"
> "testing"
> )
>
> var Foo1 []string
> var Foo2 []string
> var Count int = 32767
>
> func Benchmark1(b *testing.B) {
> for i := 0; i < b.N; i++{
> Foo1 = foo(Count)
> }
> }
>
> func Benchmark2(b *testing.B) {
> for i := 0; i < b.N; i++{
> Foo1 = foo(Count)
> Foo2 = foo(Count)
> }
> }
>
> func Benchmark3(b *testing.B) {
> for i := 0; i < b.N; i++{
> b.StopTimer()
> Foo1 = foo(Count)
> b.StartTimer()
> Foo2 = foo(Count)
> }
> }
>
> func foo(count int) []string{
> testData := []string{}
> for i:= 0; i < count; i++ {
> testData = append(testData, strconv.Itoa(i))
> }
>
> return testData
> }
>
>
> When the benchmarks are run the results are as follows:
>
> Benchmark1-4 351 3345215 ns/op
> Benchmark2-4 166 7206582 ns/op
> Benchmark3-4 334 3457907 ns/op
> PASS
> ok bar.com/benchmarks 6.881s
>
> OK benchmark3 is a little slower than Benchmark1 but that's not what's
> bothering me. It's this: if I now change Count to something much smaller
> the results are a surprise, at least to me. Here are the results when Count
> = 8:
>
> Benchmark1-4 2706196 442 ns/op
> Benchmark2-4 1357482 873 ns/op
> Benchmark3-4 840729 1387 ns/op
> PASS
> ok bar.com/benchmarks 23.547s
>
> The ratio of timings for Benchmark1 and Benchmark2 are roughly in line
> with expectations but I was surprised to see that the timings for
> Benchmark3 are now larger than those for Benchmark2.
>
> Can anyone explain this?
>
> TIA
> Orson
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golan...@googlegroups.com <javascript:>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/38f10940-a770-4fc5-86a9-19cc1371152a%40googlegroups.com
>
> <https://groups.google.com/d/msgid/golang-nuts/38f10940-a770-4fc5-86a9-19cc1371152a%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/1d291cef-420a-4f80-b719-e0654585c25d%40googlegroups.com.
