hi Areg,
On Fri, Mar 29, 2019 at 1:25 AM Melik-Adamyan, Areg
<areg.melik-adam...@intel.com> wrote:
>
> Back to the benchmarking per commit.
>
> So currently I have fired a community TeamCity Edition here
> http://arrow-publi-1wwtu5dnaytn9-2060566241.us-east-1.elb.amazonaws.com and
> dedicated pool of two Skylake bare metal machines (Intel(R) Core(TM) i7-6700
> CPU @ 3.40GHz) This can go to up to 4 if needed.
> Then the machines are prepared for benchmarking in the following way:
> - In BIOS/Setup power saving features are disabled
> - Machines are locked for access using pam_access
> - Max frequency is set through cpupower and in /etc/sysconfig/cpupower
> - All services that are not needed switched off: > uptime 23:15:17 up 26
> days, 23:24, 1 user, load average: 0.00, 0.00, 0.00
> - Transparent huge pages set on demand cat
> /sys/kernel/mm/transparent_hugepage/enabled
> always [madvise] never
> - audit control switched off auditctl -e 0
> - Memory clean added to launch scripts echo 3 > /proc/sys/vm/drop_caches
> - pstate=disable added to the kernel config
>
> This config is giving relatively clean and not noisy machine.
> Commits in master trigger build and ctest -L benchmarks. Output is parsed.
When you say "output is parsed", how is that exactly? We don't have
any scripts in the repository to do this yet (I have some comments on
this below). We also have to collect machine information and insert
that into the database. From my perspective we have quite a bit of
engineering work on this topic ("benchmark execution and data
collection") to do.
My team and I have some physical hardware (including an Aarch64 Jetson
TX2 machine, might be interesting to see what the ARM64 results look
like) where we'd like to run benchmarks and upload the results also,
so we need to write some documentation about how to add a new machine
and set up a cron job of some kind
>
> What is missing:
> * Where should our Codespeed database reside? I can fire-up a VM and put it
> there, or if you have other preferences let's discuss.
Since this isn't ASF-owned infrastructure, it can go anywhere. It
would be nice to make backups publicly available
> * What address should it have?
The address can be anything really
> * How to make it available to all developers? Do we want to integrate into CI
> or not?
I'd like to eventually have a bot that we can ask to run a benchmark
comparison versus master. Reporting on all PRs automatically might be
quite a bit of work (and load on the machines)
> * What is the standard benchmark output? I suppose Googlebench, but lets
> state that.
I thought the idea (based on our past e-mail discussions) was that we
would implement benchmark collectors (as programs in the Arrow git
repository) for each benchmarking framework, starting with gbenchmark
and expanding to include ASV (for Python) and then others
> * My interest is the C++ benchmarks only for now. Do we need to track all
> benchmarks?
Yes I think we want to be able to run the Python benchmarks too and
insert that data. Other languages can implement a benchmark collector
to arrange their benchmark data according to the database schema
> * What is the process of adding benchmarks?
Normal pull requests (see all the C++ programs that end in
"-benchmark.cc"). The benchmark collector / insertion scripts may need
to recognize when a benchmark has been run for the first time (I
haven't looked closely enough at the schema to see if there are any
primary keys associated with a particular benchmark name)
>
> Anything else for short term?
It seems like writing the benchmark collector script that runs the
benchmarks, collects machine information, and inserts data into an
instance of the database is the next milestone. Until that's done it
seems difficult to do much else
>
> -Areg.
>
>
>
>
>
