Hello, A small status update, I recently implemented archery [1] a tool for Arrow benchmarks comparison [2]. The documentation ([3] and [4]) is in the pull-request. The primary goal is to compare 2 commits (and/or build directories) for performance regressions. For now, it supports C++ benchmarks. This is accessible via the command `archery benchmark diff`. The end result is a one comparison per line, with an regression indicator.
Currently, there is no facility to perform a single "run", e.g. run benchmarks in the current workspace without comparing to a previous version. This was initially implemented in [5] but depended heavily on ctest (with no control on execution). Once [1] is merged, I'll re-implement single run (ARROW-5071) this in term of archery, since it already execute and parses C++ benchmarks. The next goal is to be able to push the results into an upstream database, be it the one defined in dev/benchmarking, or codespeed as Areg proposed. The steps required for this: - ARROW-5071: Run and format benchmark results for upstream consumption (ideally under the `archery benchmark run` sub-command) - ARROW-5175: Make a list of benchmarks to include in regression checks - ARROW-4716: Collect machine and benchmarks context - ARROW-TBD: Push benchmark results to upstream database In parallel, with ARROW-4827, Krisztian and I are working on 2 related buildbot sub-projects enabling some regression detection: - Triggering on-demand benchmark comparison via comments in PR (as proposed by Wes) - Regression check on master merge (without database support) François P.S. A collateral of this PR is that archery is a modular python library and can be used for other purposes, e.g. it could centralize orphaned scripts in dev/, e.g. linting, release, and merge since it offers utilities to handle arrow sources, git, cmake and exposes a usable CLI interface (with documentation). [1] https://github.com/apache/arrow/pull/4141 [2] https://jira.apache.org/jira/browse/ARROW-4827 [3] https://github.com/apache/arrow/blob/512ae64bc074a0b620966131f9338d4a1eed2356/docs/source/developers/benchmarks.rst [4] https://github.com/apache/arrow/pull/4141/files#diff-7a8805436a6884ddf74fe3eaec697e71R216 [5] https://github.com/apache/arrow/pull/4077 On Fri, Mar 29, 2019 at 3:21 PM Melik-Adamyan, Areg < areg.melik-adam...@intel.com> wrote: > >When you say "output is parsed", how is that exactly? We don't have any > scripts in the repository to do this yet (I have some comments on this > below). We also have to collect machine information and insert that into > the database. From my >perspective we have quite a bit of engineering work > on this topic ("benchmark execution and data collection") to do. > Yes I wrote one as a test. Then it can do POST to the needed endpoint the > JSON structure. Everything else will be done in the > > >My team and I have some physical hardware (including an Aarch64 Jetson > TX2 machine, might be interesting to see what the ARM64 results look like) > where we'd like to run benchmarks and upload the results also, so we need > to write some documentation about how to add a new machine and set up a > cron job of some kind. > If it can run Linux, then we can setup it. > > >I'd like to eventually have a bot that we can ask to run a benchmark > comparison versus master. Reporting on all PRs automatically might be quite > a bit of work (and load on the machines) > You should be able to choose the comparison between any two points: > master-PR, master now - master yesterday, etc. > > >I thought the idea (based on our past e-mail discussions) was that we > would implement benchmark collectors (as programs in the Arrow git > repository) for each benchmarking framework, starting with gbenchmark and > expanding to include ASV (for Python) and then others > I'll open a PR and happy to put it into Arrow. > > >It seems like writing the benchmark collector script that runs the > benchmarks, collects machine information, and inserts data into an instance > of the database is the next milestone. Until that's done it seems difficult > to do much else > Ok, will update the Jira 5070 and link the 5071. > > Thanks. >
