This info is really useful for other languages that want to publish to conbench
also, I'd love to find some time to try to also publish the Go benchmarks there
too, luckily the go impl is already in the main arrow repo, so other than
figuring out how to do the publishing, I'd just need Wes' blessing.
If I manage to find some time, I'll probably look into this more seriously for
the Go implementation. Would it make sense to have something actually published
somewhere that is more updated since the blog post is outdated now?
-----Original Message-----
From: Diana Clarke <diana.joan.cla...@gmail.com>
Sent: Saturday, September 11, 2021 11:37 AM
To: dev@arrow.apache.org
Subject: Re: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
I'd love to see arrow-rs and arrow-datafusion also publish results to Arrow's
Conbench server.
If you point me to the existing benchmarks for each project and instructions on
how to execute them, I can let you know the easiest integration path.
If the arrow-rs benchmarks are executable from command line and return parsable
results (like json), it should be pretty easy to publish the results.
And if any of the existing benchmarks are written in Python (perhaps
arrow-datafusion?), then there may be a different (but also easy) integration
path.
There are a few caveats though:
- The benchmarks are running on dedicated machines at Wes' house, so we would
need his blessing (or additional dedicated machines).
- The arrow-rs and arrow-datafusion GitHub repositories must use squash merges
(or Conbench would have to be extended to understand the
2 other GitHub merge methods).
- I'm not sure what the security implications are with respect to adding our
ursabot integration and buildkite hooks to other repositories.
In the meantime, here's a blog post that initially announced the Arrow/Conbench
integration and contains a brief functionality overview.
https://urldefense.com/v3/__https://ursalabs.org/blog/announcing-conbench/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SishwuMxk$
That blog post is a bit outdated now – we've since started publishing the Arrow
Java and JavaScript benchmarks to
https://urldefense.com/v3/__https://conbench.ursa.dev/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiMWTIttY$
as well.
And Conbench now collects and plots benchmark distribution history, and
computes z-scores (rather than just a basic percent changed).
Example distribution history (note the regression):
https://urldefense.com/v3/__https://conbench.ursa.dev/compare/benchmarks/7efda5a661a348ebb66e021dc9aa7bba...6a4122865bba4b6b9045390d4d1eb7cc/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiHXS6EbM$
Example simple benchmark result plotting:
https://urldefense.com/v3/__https://conbench.ursa.dev/batches/cbe57f13e51f46019f207853421db764/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SinbmHAmQ$
Example run of the Arrow C++ and Java benchmarks:
https://urldefense.com/v3/__https://conbench.ursa.dev/runs/82308256f17a4d059db902b07d4d4530/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiXPdw6A4$
Example run of the Arrow R, Python, and JavaScript benchmarks:
https://urldefense.com/v3/__https://conbench.ursa.dev/runs/b56fdd1375ed4b0cbf928dd8dafe8322/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiWBdn2lw$
Finally, here's a link to the Conbench repository (as well as the repositories
with the Arrow benchmarks):
- Conbench:
https://urldefense.com/v3/__https://github.com/ursacomputing/conbench__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiTUlgkB0$
- Arrow Python benchmarks + external benchmark wrappers/runners:
https://urldefense.com/v3/__https://github.com/ursacomputing/benchmarks__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SifALMAmM$
- Arrow R benchmarks:
https://urldefense.com/v3/__https://github.com/ursacomputing/arrowbench__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiRrJfhbg$
- Arrow Java/JavaScript/C++ benchmarks:
https://urldefense.com/v3/__https://github.com/apache/arrow__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8Sizr5ywCE$
Looking forward to collaborating on this!
--diana
On Sat, Sep 11, 2021 at 5:01 AM Andrew Lamb <al...@influxdata.com> wrote:
>
> I wasn't even aware of conbench.ursa.dev -- I agree it looks like a
> really neat tool and could be very valuable for the rust language
> implementation
>
> On Fri, Sep 10, 2021 at 11:16 PM QP Hou <ho...@apache.org> wrote:
>
> > Hi,
> >
> > I think conbench.ursa.dev works really well for the main arrow repo,
> > especially the ability to request on demand benchmarks during PR
> > reviews by mentioning usrabot in a comment.
> >
> > I am wondering if it is something that arrow-rs and arrow-datafusion
> > could leverage as well to help speed up PR reviews and guard against
> > performance degradation? If so, what's the process for setting it up?
> >
> > Thanks,
> > QP
> >
