serhiy-bzhezytskyy opened a new pull request, #19:
URL: https://github.com/apache/solr-orbit-workloads/pull/19
Follow-up to #18, which fixed the `range` operation. The same rule was never
applied to the other four nyc_taxis operations that carry a range bound, and
upstream writes an exclusive upper bound in every one of them:
| operation | upstream bound | port before | port after |
|---|---|---|---|
| `distance_amount_facet` | `lt: 50` | `trip_distance:[0 TO 50]` |
`trip_distance:[0 TO 50}` |
| `date_histogram_calendar_interval` | `lt: 2016-01-01 00:00:00` |
`...2016-01-01T00:00:00Z]` | `...Z}` |
| `date_histogram_fixed_interval` | `lt: 2016-01-01 00:00:00` |
`...2016-01-01T00:00:00Z]` | `...Z}` |
| `date_histogram_facet` | `lte: 21/01/2015`, `dd/MM/yyyy` |
`...2015-01-21T00:00:00Z]` | `...2015-01-22T00:00:00Z}` |
`date_histogram_facet` is not just a bracket. A date-only bound in
OpenSearch names
a whole day and is rounded: `lte` goes to that day's last millisecond. So
upstream
covers the whole of 21 January, while the port cut the filter at the first
instant
of the 21st and lost a day.
The file also disagreed with itself: that operation's own facet already runs
to
`end: 2015-01-22T00:00:00Z` with `gap: +1DAY`, asking for 21 daily buckets,
and
with `mincount: 1` the last one could never be populated.
## Measured
300,649 documents loaded into Solr 10.0.0 and OpenSearch 3.8.0 **from one
file**,
line number as the id on both sides. Provenance checked rather than assumed:
300,649 corpus lines against 300,649 OpenSearch documents, 10 sampled ids
byte-identical, and the Solr schema read back from the live core.
| operation | OpenSearch | before | after |
|---|---:|---:|---:|
| `date_histogram_facet` | **17,651** | 16,811 | **17,651** |
| `distance_amount_facet` | 300,630 | 300,630 | 300,630 |
| `date_histogram_calendar_interval` | 300,640 | 300,640 | 300,640 |
Equal counts can hide compensating differences, so for
`date_histogram_facet` I
compared document id *sets*, not just totals: before, 840 missing and 0
extra;
after, the sets are equal. And the buckets the operation actually returns:
```
OpenSearch 21 buckets last: ('2015-01-21', 840)
before 20 buckets last: ('2015-01-20', 828)
after 21 buckets last: ('2015-01-21', 840)
```
`2015-01-21` is absent from the old output entirely.
## What this does not show
Only the first change is observable on this corpus. Nothing in the sample has
`trip_distance` of exactly 50, or a dropoff at exactly
`2016-01-01T00:00:00Z`, so
the other three bounds are **correct by the rule and latent by the data** —
which
is the reason to fix them from upstream's operator rather than from a
measurement.
## Related
`convert-workload` reproduces two of these independently
(apache/solr-orbit#68 and
the date-conversion issue filed alongside it), so regenerating the workload
would
reintroduce them. Fixing the shipped file here does not depend on those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
