serhiy-bzhezytskyy opened a new pull request, #74:
URL: https://github.com/apache/solr-orbit/pull/74
Closes #73.
Two date-bound defects in `_convert_date_to_solr_format`, both independent
of the
bracket fix in #69 and both still present on `main` at f0883807.
**A datetime with a space separator was passed through verbatim.**
`yyyy-MM-dd HH:mm:ss`
was missing from the format map, and an unknown format is logged and used
as-is. A space
inside a range term ends the term, so Solr answers HTTP 400 and the
operation cannot run
at all:
```
Cannot parse 'dropoff_datetime:[2015-01-01 00:00:00 TO 2016-01-01 00:00:00}':
Encountered " <RANGE_GOOP> "00:00:00 "" at line 1, column 28.
```
#69 gave that query a correct exclusive bracket, which does not help while
the query
still fails to parse. Two nyc_taxis operations are affected.
**A whole-day `lte` was not rounded.** A bound with no time names a *day*,
and OpenSearch
rounds it to that day's edge — `lte` and `gt` to its last millisecond, `gte`
and `lt` to
its first. Solr rounds nothing, so only the two that move to the end of the
day need
translating. `dd/MM/yyyy` was already in the map, so the value converted
cleanly and
silently landed on midnight, dropping the rest of the day.
Note the bracket in that case is inclusive on `main` and stays inclusive
here: upstream
writes `lte`, so `]` was always right. The only thing wrong was the value.
## Approach
The format map now records, per format, whether the pattern carries a time,
so a
whole-day bound is recognisable after parsing rather than guessed at from
its length.
Rounding then advances such a bound to the following day's first instant and
makes the
bracket exclusive:
```
dropoff_datetime:[2015-01-01T00:00:00Z TO 2015-01-22T00:00:00Z}
```
`...T23:59:59.999Z]` would be the more literal reading, but it assumes
Solr's date
precision is exactly a millisecond. Naming the next day's first instant says
"the whole
of the 21st" without depending on that.
## Verification
300,649 nyc_taxis documents indexed into Solr 10.0.0 and OpenSearch 3.8.0
**from one
file**, line number as the id on both sides. Provenance checked rather than
assumed:
300,649 corpus lines against 300,649 OpenSearch documents, 10 sampled ids
byte-identical,
and the Solr schema read back from the live core.
| | OpenSearch | before | after |
|---|---:|---:|---:|
| `date_histogram_facet` matches | **17,651** | 16,811 | **17,651** |
| daily buckets returned | 21 | 20 | 21 |
Equal counts can hide compensating differences, so I compared document id
*sets* rather
than totals: before, 840 missing and 0 extra; after, the sets are equal. And
the buckets
the operation actually returns:
```
OpenSearch 21 buckets last: ('2015-01-21', 840)
before 20 buckets last: ('2015-01-20', 828)
after 21 buckets last: ('2015-01-21', 840)
```
`2015-01-21` is absent from the old output entirely, not merely short.
Separately, the converter's output for the three real upstream nyc_taxis
bodies is now
byte-identical to what the hand-written Solr port ships — two independent
routes reaching
the same string.
5 new unit tests; `pytest tests/` passes 1115, skips 5.
apache/solr-orbit-workloads#19 corrected the same bounds in the checked-in
`nyc_taxis`
files. This is the generator, so without it a regeneration reintroduces both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
