janhoy opened a new pull request, #109:
URL: https://github.com/apache/solr-orbit/pull/109
## What this is
`reportr` — the benchmark trends website from #99, phase 1: everything
local, no cloud. It reads the enriched reports `launchr` writes under
`./benchmark-results/` and emits a self-contained static site: one page per
workload, charts of every metric over Solr's own version timeline, and the full
report behind any point one click away.
```
reportr build --local ./benchmark-results --serve
```
That is the whole workflow. No S3, no registry polling, no services — the
output is a directory of files that works from any file server.
> **This stacks on #108 — please review that one first.** It is opened
against `main` only because GitHub requires a PR's base branch to live in the
base repository, and `launchr-phase1-local-mvp` currently exists only in the
fork; the base will be retargeted to it. Until then the diff carries #108's two
commits as well — `Add launchr, the benchmark orchestrator` and `Let
SolrDockerLauncher use a configurable image repository`. Everything from `Add
reportr, the benchmark trends site generator` onward is this PR: nine commits.
## The decisions worth arguing about
**The x-axis is Solr's history, not the benchmark's.** A point sits at the
date its Solr version was created, not the date the benchmark ran, so
re-benchmarking an old release lands it in its correct historical position
rather than at the right-hand edge. The version is labelled on the axis, as an
upright chip under the date ticks — once per build rather than once per dot,
since every line crossing that date ran the same one.
**Not every point is a release.** A release is named by its version. A
snapshot is named by where it came from and what it was built at —
`9x#a1b2c3d`, `main-nightly` — because a nightly's version string is identical
for every build on that branch. The word "release" appears in the UI only when
every line on the page is one.
**A line is one version kind, branch, profile and cluster config.** That is
the whole identity, deliberately excluding the run id, the hostname and the
workload revision: a workload change gets an annotation rather than a new line,
so the long-run trend stays continuous. It means the site will not silently
compare a 3-node EC2 result against a laptop run.
**A query is charted on what it was driven for.** Solr Orbit now records
each task's target throughput with the result, so reportr can tell a query held
at three requests a second from one pushed as hard as it would go. The first
says nothing through its throughput — that number is the rate that was asked
for — so it is read on latency. The second is read on queries per second, and
leads its section.
**Query latency charts lead with p50 and p90.** The stock query tasks
measure 100 iterations after warmup, and the 99th percentile of 100 samples is
the second-highest observation — a single draw that moves run to run for
reasons unrelated to Solr. p99 is still charted when a run carries it, because
the tail is what users feel. It is just not the headline.
**A band shows what the headline number came out of** — the slowest and
fastest sampling interval behind a mean throughput, the median and the tail
behind a p90. Solr Orbit reports these and nothing was drawing them, so a mean
whose intervals ranged 3k–9k docs/s looked exactly as settled as one where
every interval hit 4.7k.
**A heatmap answers "what moved, and when?"** — one row per task, one column
per build, signed so positive always means better. A page of line charts
answers that question only by making you scan every one of them.
**Nothing is inferred to be a regression.** What a step means is a
judgement, and a detector that cried wolf would be worse than no annotation, so
markers come from three places that each know something the numbers do not: the
workload revision changed, a task's own `reportr.note` changed, or somebody
wrote a release note in `reportr.yaml`.
**Workloads describe and curate themselves.** `workload.json` has had a
`description` field all along; Solr Orbit now carries it into the result the
way it already carries `workload-revision`. Everything else a workload wants to
say goes in the `meta` block Solr Orbit already merges onto every operation:
```json
"meta": { "reportr.description": "165 million rides, and what they
exercise." }
{ "name": "range", "operation": "range", "meta": {
"reportr.chart": "own",
"reportr.note": "json.facet, not the legacy facet component" } }
```
Editing `reportr.note` is the workload saying the thing it measures is no
longer quite the same thing, and the chart marks the point it happened.
`reportr.yaml` overrides all of it per workload (`description`, `tasks`,
`default_tasks`, `featured_tasks`) and rejects a misspelled key rather than
silently ignoring it. The workload suggests; whoever runs the site decides.
## Reading the page
Chips are the tasks the run measured, and nothing else, and a chip makes one
promise: this task gets a chart of its own, inside the section for the kind of
work it does — `range` is one of the queries, not a peer of `Query`.
`featured_tasks` is the initial selection, not a ceiling; any task can join or
leave it. Queries lead the page and indexing closes it.
Each chart's legend carries every task of its kind and merely opens with
some of them, so comparing one query against its neighbour never means leaving
the chart. Sections fold from the caret and stay folded on the next visit.
Every section and every task chart has a permalink, and a link produces what it
points at — switching a task on and unfolding its section if it has to, since
the URL carries none of the reader's own selection.
## Changes outside reportr
Two, both in `solrorbit/metrics.py`, both additive:
- `TestRun` gained `workload_description`, taken from the workload's own
`description`, written to `as_dict()` and each result record as
`workload-description`, round-tripping through `from_dict`. A workload without
one omits the key rather than storing an empty string.
- `op_metrics` entries gained a `schedule` block — target throughput and
unit, clients, warmup and measurement windows — so a consumer reading stored
JSON can tell how a task was driven.
## Reviewing it
- `models.py`, `extract.py`, `series.py` — the data model, and what is
deliberately tolerated (a missing percentile) versus refused (a
`schema_version` newer than this code understands).
- `builder.py`, `config.py` — what lands in the site, and how a workload is
curated.
- `site/` — the page. Vanilla HTML/CSS/JS plus a vendored Apache ECharts
5.5.1 (Apache-2.0). Nothing is fetched from a CDN at runtime, and there is a
test asserting that.
179 reportr tests, 1610 in the suite, ruff clean.
## What is not here
Cloud storage, incremental builds, retention, and deployment to
`nightlies.apache.org` are later phases of #99. So are the dashboard-overview
and cluster-topology ideas from the mockup.
Refs #99
🤖 Assisted by Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
