I'm wondering if it would be useful to build benchmark regression detection into Go test benchmarks, and policy enforcement in terms of making regressions fail a benchmark "test".
If I understand correctly, as Go benchmarks currently stand, they tell you the numbers, but it's left to you to interpret them, store them, compare them, and enforce policies around them. For Go users who aren't performance and benchmarking experts, it would be useful to build in a simple process for doing this. Perhaps benchmark "bars" could be stored under $GOPATH like fuzz data, and a benchmark bar could be specified or saved in the package itself. This would be useful for any project that had a policy of no performance regressions, or only permitting a certain amount of regression, such as a browser renderer <https://nicoverbruggen.be/blog/webkit-no-regression-policy>. The Go Team could use it to track Go performance and enforce its own regression policies for Go changes. Perhaps something like this: $ pwd /proj $ ls go.mod go.sum proj.go proj_test.go $ go test -bench . # result saved under $GOPATH PASS BenchmarkThing 5000000 500 ns/op ok github.com/my/proj 3.0s $ vi proj.go # make changes $ go test -bench . # much slower PASS, REGRESSION BenchmarkThing 5000000 900 ns/op ok github.com/my/proj 3.1s $ vi proj.go # revert changes $ go test -bench . # much faster PASS, REGRESSION FIXED BenchmarkThing 5000000 501 ns/op ok github.com/my/proj 3.0s $ ls go.mod go.sum proj.go proj_test.go $ go test -bench . -benchsave # save benchmark bar to go.bench PASS BenchmarkThing 5000000 502 ns/op ok github.com/my/proj 3.0s $ ls go.bench go.mod go.sum proj.go proj_test.go $ vi proj.go # make changes $ go clean -benchcache # delete results saved under $GOPATH $ go test -bench . # compares to go.bench FAIL, REGRESSION BenchmarkThing 5000000 901 ns/op ok github.com/my/proj 3.1s Will -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAKbcuKiwdCsJOXTWoEQar_aSHfCk4H1yjvBkDimeMxfLb5-rYA%40mail.gmail.com.
