In the benchmarking one of the hardest parts (IMHO) is the process/workflow automation. I'm in support of the development of a "meta-benchmarking" framework that offers automation, extensibility, and possibility for customization.
One of the reasons that people don't do more benchmarking as part of their development process is that the tooling around it isn't great. Using a command line tool [1] that outputs unconfigurable text to the terminal to compare benchmarks seems inadequate to me. In the cited example $ benchcmp old.txt new.txt Where do old.txt and new.txt come from? I would like to have that detail (build of appropriate component, execution of benchmarks and collection of results) automated. FWIW, 7 and a half years ago [2] I wrote a small project called vbench to assist with benchmark automation, so this has been a long-term interest of mine. Codespeed existed in 2011, here is what I wrote about it in December 2011, and it is definitely odd to find myself typing almost the exact same words years later: "Before starting to write a new project I looked briefly at codespeed... The dealbreaker is that codespeed is just a web application. It doesn't actually (to my knowledge, someone correct me if I'm wrong?) have any kind of a framework for orchestrating the running of benchmarks throughout your code history." asv [3] is a more modern and evolved version of vbench. But it's Python-specific. I think we need the same kind of thing except being able to automate the execution of any benchmarks for any component in the Arrow project. So we have some work to do. - Wes [1]: https://github.com/golang/tools/blob/master/cmd/benchcmp/benchcmp.go [2]: http://wesmckinney.com/blog/introducing-vbench-new-code-performance-analysis-and-monitoring-tool/ [3]: https://github.com/airspeed-velocity/asv On Wed, Apr 24, 2019 at 11:18 AM Sebastien Binet <bi...@cern.ch> wrote: > > On Wed, Apr 24, 2019 at 11:22 AM Antoine Pitrou <anto...@python.org> wrote: > > > > > Hi Areg, > > > > Le 23/04/2019 à 23:43, Melik-Adamyan, Areg a écrit : > > > Because we are using Google Benchmark, which has specific format there > > is a tool called becnhcmp which compares two runs: > > > > > > $ benchcmp old.txt new.txt > > > benchmark old ns/op new ns/op delta > > > BenchmarkConcat 523 68.6 -86.88% > > > > > > So the comparison part is done and there is no need to create infra for > > that. > > > > "surprisingly" Go is already using that benchmark format :) > and (on top of a Go-based benchcmp command) there is also a benchstat > command that, given a set of multiple before/after data points adds some > amount of statistical analysis: > https://godoc.org/golang.org/x/perf/cmd/benchstat > > using the "benchmark" file format of benchcmp and benchstat would allow > better cross-language interop. > > cheers, > -s
