[
https://issues.apache.org/jira/browse/SOLR-18418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18418:
----------------------------------
Labels: pull-request-available (was: )
> complement() and intersect() silently return wrong results when on= maps
> different field names
> ----------------------------------------------------------------------------------------------
>
> Key: SOLR-18418
> URL: https://issues.apache.org/jira/browse/SOLR-18418
> Project: Solr
> Issue Type: Bug
> Components: streaming expressions
> Reporter: David Smiley
> Assignee: David Smiley
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{complement()}} and {{intersect()}} silently return incorrect results when
> the {{on=}} clause maps two *different* field names (e.g.
> {{on="parent_id=id"}}).
> There is no exception and no warning. {{complement()}} returns its *entire*
> input (every tuple reported as non-matching) and {{intersect()}} returns
> *nothing*. The output is superficially believable, so a wrong conclusion is
> easy to reach and hard to notice.
> h2. Toy example
> Stream A has field {{parent_id}} with values {{a, b}}. Stream B has field
> {{id}} with values {{b, c}}. Both are sorted ascending.
> {noformat}
> complement(A, B, on="parent_id=id")
> expected: a
> actual: a, b <-- "b" clearly exists in B
> intersect(A, B, on="parent_id=id")
> expected: b
> actual: (nothing)
> {noformat}
> The same query is correct if the join field has the *same name* in both
> streams.
> h2. Conditions to reproduce
> # an asymmetric {{on="x=y"}} (different field names on each side), and
> # a first A-value that is *absent* from B — {{a}} in the example above.
> If the first A-value happens to match, results look fine, which is why small
> or fully-matching test data does not expose this.
> Observed on a ~4.1M row collection: {{complement()}} reported 100% of rows as
> non-matching. The true figure was ~26%. The tell is that the output count
> exactly equalled the input count.
> h2. Cause (brief)
> {{ComplementStream:204}} and {{IntersectStream:212}} do the cross-stream
> ordering comparison with {{streamA.getStreamSort()}}, whose left and right
> field names are *both* stream A's sort field. Applied to a tuple from stream
> B it reads a field B does not have, gets {{null}}, and returns a constant —
> so the comparison never reports "less than". The {{on=}} mapping is honoured
> only by {{eq.test()}}, never by the comparison that drives the merge.
> Consequence: the first unmatched A-value scans stream B all the way to EOF,
> and B is forward-only. Every subsequent tuple then falls straight through as
> non-matching.
> h2. Notes
> * The guard at {{ComplementStream:102}} is meant to reject asymmetric
> {{on=}}, but {{FieldEqualitor.isDerivedFrom(StreamComparator)}} combines its
> two field checks with {{||}}, so it passes.
> * {{innerJoin}}, {{leftOuterJoin}} and {{fullOuterJoin}} handle asymmetric
> {{on=}} correctly — they derive the comparison from the equalitor rather than
> from one stream's sort. {{complement}} and {{intersect}} are the only two
> decorators that compare across streams using {{getStreamSort()}}.
> * Workaround: alias one side so both streams use the same field name, e.g.
> wrap A in {{select(A, parent_id as id)}} and join with {{on="id"}}.
> * Worth asserting in any analysis: {{complement}} and {{intersect}} partition
> the input, so their counts must sum to {{|A|}}.
> * Both lines were last touched only by the 2022 Spotless reformat
> (SOLR-14920), so this appears to be long-standing.
> _obviously written by AI_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
