David Smiley created SOLR-18418:
-----------------------------------
Summary: complement() and intersect() silently return wrong
results when on= maps different field names
Key: SOLR-18418
URL: https://issues.apache.org/jira/browse/SOLR-18418
Project: Solr
Issue Type: Bug
Components: streaming expressions
Reporter: David Smiley
Assignee: David Smiley
{{complement()}} and {{intersect()}} silently return incorrect results when the
{{on=}} clause maps two *different* field names (e.g. {{on="parent_id=id"}}).
There is no exception and no warning. {{complement()}} returns its *entire*
input (every tuple reported as non-matching) and {{intersect()}} returns
*nothing*. The output is superficially believable, so a wrong conclusion is
easy to reach and hard to notice.
h2. Toy example
Stream A has field {{parent_id}} with values {{a, b}}. Stream B has field
{{id}} with values {{b, c}}. Both are sorted ascending.
{noformat}
complement(A, B, on="parent_id=id")
expected: a
actual: a, b <-- "b" clearly exists in B
intersect(A, B, on="parent_id=id")
expected: b
actual: (nothing)
{noformat}
The same query is correct if the join field has the *same name* in both streams.
h2. Conditions to reproduce
# an asymmetric {{on="x=y"}} (different field names on each side), and
# a first A-value that is *absent* from B — {{a}} in the example above.
If the first A-value happens to match, results look fine, which is why small or
fully-matching test data does not expose this.
Observed on a ~4.1M row collection: {{complement()}} reported 100% of rows as
non-matching. The true figure was ~26%. The tell is that the output count
exactly equalled the input count.
h2. Cause (brief)
{{ComplementStream:204}} and {{IntersectStream:212}} do the cross-stream
ordering comparison with {{streamA.getStreamSort()}}, whose left and right
field names are *both* stream A's sort field. Applied to a tuple from stream B
it reads a field B does not have, gets {{null}}, and returns a constant — so
the comparison never reports "less than". The {{on=}} mapping is honoured only
by {{eq.test()}}, never by the comparison that drives the merge.
Consequence: the first unmatched A-value scans stream B all the way to EOF, and
B is forward-only. Every subsequent tuple then falls straight through as
non-matching.
h2. Notes
* The guard at {{ComplementStream:102}} is meant to reject asymmetric {{on=}},
but {{FieldEqualitor.isDerivedFrom(StreamComparator)}} combines its two field
checks with {{||}}, so it passes.
* {{innerJoin}}, {{leftOuterJoin}} and {{fullOuterJoin}} handle asymmetric
{{on=}} correctly — they derive the comparison from the equalitor rather than
from one stream's sort. {{complement}} and {{intersect}} are the only two
decorators that compare across streams using {{getStreamSort()}}.
* Workaround: alias one side so both streams use the same field name, e.g. wrap
A in {{select(A, parent_id as id)}} and join with {{on="id"}}.
* Worth asserting in any analysis: {{complement}} and {{intersect}} partition
the input, so their counts must sum to {{|A|}}.
* Both lines were last touched only by the 2022 Spotless reformat (SOLR-14920),
so this appears to be long-standing.
_obviously written by AI_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
