On Thu, Apr 25, 2019 at 1:28 AM Melik-Adamyan, Areg
wrote:
>
> Hi,
>
> We are talking about the same thing actually, but you do not want to use 3rd
> party tools.
> For 3 and 4 - you run the first version store in 1.out, then second version
> store in 2.out and run compare tool. Your tool does t
Hi,
We are talking about the same thing actually, but you do not want to use 3rd
party tools.
For 3 and 4 - you run the first version store in 1.out, then second version
store in 2.out and run compare tool. Your tool does two steps automatically,
that is fine.
> Various reason why I think th
Hello,
archery is the "shim" scripts that glue some of the steps (2-4) that you
described. It builds arrow (c++ for now), find the multiple benchmark
binaries, runs them, and collects the outputs. I encourage you to check the
implementation, notably [1] and [2] (and generally [3]).
Think of it as
Wes,
The process as I think should be the following.
1. Commit triggers to build in TeamCity. I have set the TeamCity, but we can
use whatever CI we would like.
2. TeamCity is using the pool of identical machines to run the predefined (or
all) performance benchmarks on one the build machines fro
mailing list thread [was Fwd: [Discuss] Benchmarking
infrastructure]
On Wed, Apr 24, 2019 at 11:22 AM Antoine Pitrou wrote:
>
> Hi Areg,
>
> Le 23/04/2019 à 23:43, Melik-Adamyan, Areg a écrit :
> > Because we are using Google Benchmark, which has specific format
> >
In the benchmarking one of the hardest parts (IMHO) is the
process/workflow automation. I'm in support of the development of a
"meta-benchmarking" framework that offers automation, extensibility,
and possibility for customization.
One of the reasons that people don't do more benchmarking as part o
On Wed, Apr 24, 2019 at 11:22 AM Antoine Pitrou wrote:
>
> Hi Areg,
>
> Le 23/04/2019 à 23:43, Melik-Adamyan, Areg a écrit :
> > Because we are using Google Benchmark, which has specific format there
> is a tool called becnhcmp which compares two runs:
> >
> > $ benchcmp old.txt new.txt
> > bench
No worries,
I'll update the PR to refactor this cli function in a re-usable function.
Luckily it's small enough and not too much logic is leaking.
On Tue, Apr 23, 2019 at 4:24 PM Wes McKinney wrote:
> hi Francois,
>
> This sounds like good progress.
>
> For any tool consumable through a CLI/com
Hi Areg,
Le 23/04/2019 à 23:43, Melik-Adamyan, Areg a écrit :
> Because we are using Google Benchmark, which has specific format there is a
> tool called becnhcmp which compares two runs:
>
> $ benchcmp old.txt new.txt
> benchmark old ns/op new ns/op delta
> BenchmarkConcat
: 29836,
"bytes_per_second": 134066,
"items_per_second": 33516
}
]
}
So we have all the ingredients and do not need to reinvent anything, we need
just to agree on the process: what is done when and put to where in which
format.
------ Forwarded mess
hi Francois,
This sounds like good progress.
For any tool consumable through a CLI/command-line interface my
recommendation is to ensure that the software is usable as a library
equally as well as via a CLI interface.
In this patch I see
https://github.com/apache/arrow/pull/4141/files#diff-7a88
Hello,
A small status update, I recently implemented archery [1] a tool for Arrow
benchmarks comparison [2]. The documentation ([3] and [4]) is in the
pull-request. The primary goal is to compare 2 commits (and/or build
directories) for performance regressions. For now, it supports C++
benchmarks.
hi David -- yes, we definitely should set up cross-host and
cross-implementation performance testing (that we can measure and
record in the benchmark database) for Flight. As one starting point
https://issues.apache.org/jira/browse/ARROW-4566
- Wes
On Mon, Apr 1, 2019 at 10:30 AM David Li wrote
One more thought, is there interest in running cross-host Flight
benchmarks, and perhaps validating them against iperf or a similar
tool? It would be great to get latency/throughput numbers and make
sure upgrades to gRPC don't tank performance on accident, and it would
help argue for why people sho
Le 29/03/2019 à 16:06, Wes McKinney a écrit :
>
>> * How to make it available to all developers? Do we want to integrate into
>> CI or not?
>
> I'd like to eventually have a bot that we can ask to run a benchmark
> comparison versus master. Reporting on all PRs automatically might be
> quite a
>When you say "output is parsed", how is that exactly? We don't have any
>scripts in the repository to do this yet (I have some comments on this below).
>We also have to collect machine information and insert that into the database.
>From my >perspective we have quite a bit of engineering work o
I see that Francois is doing some work related to this in
https://github.com/apache/arrow/pull/4077
On Fri, Mar 29, 2019 at 11:20 AM Wes McKinney wrote:
>
> hi,
>
> After doing a little research I took a closer look at the shell scripts in
>
> https://github.com/apache/arrow/tree/master/dev/benc
hi,
After doing a little research I took a closer look at the shell scripts in
https://github.com/apache/arrow/tree/master/dev/benchmarking
While these may work for importing the gbenchmark data, the general
approach seems inflexible to me, and I would recommend rewriting them
as Python programs
hi Areg,
On Fri, Mar 29, 2019 at 1:25 AM Melik-Adamyan, Areg
wrote:
>
> Back to the benchmarking per commit.
>
> So currently I have fired a community TeamCity Edition here
> http://arrow-publi-1wwtu5dnaytn9-2060566241.us-east-1.elb.amazonaws.com and
> dedicated pool of two Skylake bare metal m
Back to the benchmarking per commit.
So currently I have fired a community TeamCity Edition here
http://arrow-publi-1wwtu5dnaytn9-2060566241.us-east-1.elb.amazonaws.com and
dedicated pool of two Skylake bare metal machines (Intel(R) Core(TM) i7-6700
CPU @ 3.40GHz) This can go to up to 4 if need
20 matches
Mail list logo
