Important things for me in conbench:
* regression detection, which is quite solid (the algorithm has been
tweaked a lot AFAIK):
https://github.com/apache/arrow/runs/84305379739
* benchmark result pages and especially the "compare with baseline run
from fork point commit", e.g.
https://conbench.arrow-dev.org/runs/457fe991307a42d786798829edbc29f9/
* the comparison pages such as
https://conbench.arrow-dev.org/compare/runs/1b61ec2a2670462f8c788da25fda3ca6...4ab7d6720b8c4e698e643f35819745c7/
; currently they are quite slow to compute and often time out
Note some UI improvements would be warranted (especially: better
filtering and/or display for benchmark results), we can discuss that too.
Regards
Antoine.
Le 27/06/2026 à 03:24, Wes McKinney a écrit :
I can update the chart to be more in line with the old one.
I haven't done a great deal of work to enhance the UI (a lot of this
development was unattended and based on a mandate to rebuild the
backend in Go and rebuild the frontend on modern web technology, but
not make substantial changes). Using modern web technology (vite /
svelte) means it is much easier to add new things and make them feel
polished and nice, so the question is really what would be useful to
have?
On Fri, Jun 26, 2026 at 7:56 PM Rok Mihevc <[email protected]> wrote:
It's night here so I appreciate the new darker palette.
As for result readability I prefer the old interactive graph because it has
classified points, trendline and bands.
old:
https://conbench.arrow-dev.org/benchmark-results/06a3edbc2d1f7dc48000633dc612769e/
new::
https://conbench-v2.arrow-dev.org/benchmarks/history/06a3edbc2d1f7dc48000633dc612769e
Any new feature that you'd point out for improved ergonomics?
Rok
On Sat, Jun 27, 2026 at 2:22 AM Wes McKinney <[email protected]> wrote:
TL;DR take a look at https://conbench-v2.arrow-dev.org.
This was written by Codex to summarize the state of things:
Hi all,
I’ve pushed the Conbench v2 work for review in the following places:
Conbench v2 app/server/docs:
https://github.com/conbench/conbench/tree/experimental-v2
Buildkite, Terraform, and CI adapter work:
https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report
Python benchmark payload work:
https://github.com/wesm/benchmarks/tree/v2-conbench-submit
R benchmark payload work:
https://github.com/wesm/arrowbench/tree/v2-conbench-payloads
There is also a live read-only evaluator running here:
https://conbench-v2.arrow-dev.org
Temporary docs are published here:
https://wesm.github.io/conbench-tmp/
The goal of this work is to provide a concrete migration path for
Conbench v2 while preserving the existing production database schema.
The evaluator is intended for review and experimentation only; it
should not modify the production database.
Please take a look at the app, docs, and workflow changes. The main
things to review are whether the new UI is useful for Arrow
maintainers, whether the benchmark reporting path is understandable,
and whether the migration approach looks practical for the existing
Arrow benchmarking workflows.
On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <[email protected]> wrote:
If you can give me access to the servers and cloud resources in question
(ie the machines that currently run the benchmarks), I can implement the
necessary code adaptations and test things working end to end, and stand up
a parallel deployment of the application against RDS to enable better
evaluation of the UI ergonomics. Perhaps we can coordinate offline and come
back to the community with a report once the implementation is closer to
being able to “throw the switch”.
On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <[email protected]> wrote:
Current architecture is not optimal or modern, but its maintenance cost
is well known and currently manageable. But I'm happy with changes that
move us to a more maintainable state and am willing to assist with the
transition.
On a personal note, I'm hesitant to sign up for maintaining a deployment
of software that's not built yet.
So I'm just curious about who "owns" Arrow's conbench deployment
if I cannot commit to it.
Rok
On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <[email protected]>
wrote:
I think the objective is to create a more modern foundation while also
improving performance. I think this looks like:
* faster, easier to develop and deploy backup (use Go or Rust to
create
static binaries: Go is good for backend services like this)
* use modern web technologies versus generating pages with Jinja
templates
* Use things like DuckDB and Parquet to scale result storage while
improving performance
* Add many more UI features to make the results most useful to
maintainers
Like I said, I’m happy to fulfill feature requests and contribute
development with agents, if it isn’t interesting I’m also fine to go
my own
way.
I think Buildkite is fine for job scheduling and management for now, I
don’t think this system currently wants to own a task queue / durable
execution state for workers, though it could grow this capability in
the
future (workers would have to poll the server for jobs to take).
On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <[email protected]>
wrote:
Hi,
I think the main question here is: what are we trying to do?
Currently, the main operational issue with conbench is the slowness
of
the web UI, due to the large database size and that it's not
normalized
(some queries take much longer than they should).
I can't speak about the maintenance / reliability aspects, though.
Regards
Antoine.
Le 04/06/2026 à 16:19, Wes McKinney a écrit :
hi all,
I saw that conbench.ursa.dev has been down and I had a need to
set up
some continuous project benchmarks, and was interested in doing
development on Conbench (well, having my agents do development on
Conbench), and was interested in the following:
1) is there interest in migrating the historical Arrow conbench
data
to a new server, has that been preserved somewhere? I'll probably
rewrite the conbench backend in Go and give it a client CLI for
submitting new data or querying old data.
2) are there other users of conbench (conbench/conbench) that
anyone
is aware of? I'd be done doing in-situ development in that
repository
or setting up a conbench-v2 project.
No particular urgency but if anyone has opinions let me know!
thanks,
Wes
