[
https://issues.apache.org/jira/browse/SOLR-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Shanin updated SOLR-18539:
-------------------------------
Description:
🤖 _AI text below_ 🤖 _(posted on behalf of Nick Shanin)_
In a distributed search sorted on a field, {{QueryComponent.mergeIds}} folds
each shard's totals into the merged response before it tests the
partial-results skip, so a shard whose documents are dropped as partial still
counts. This was confirmed by a run on unmodified main with fabricated shard
responses: the merged list held 3 documents, but {{numFound}} was 5,
{{numFoundExact}} was true, {{maxScore}} was 99.0 from the skipped shard, and
{{approximateTotalHits}} was 11 instead of 4. A second defect sits in the same
code: when a shard sends no {{approximateTotalHits}} header, the fallback adds
the running merged total instead of that shard's own count, so two shards with
2 and 3 documents merged to an approximate total of 2 instead of 5.
The fix I recommend is to test the skip before accumulating, so a skipped shard
never enters the totals; subtracting its contribution afterwards cannot work,
because a maximum such as {{maxScore}} cannot be decremented and the exactness
flag would need recomputation. Setting {{hitCountIsExact}} to false when a
shard is skipped would fix only the exactness symptom and leave the counts
wrong. The fallback should also add the shard's own count instead of the
running total, as part of the same fix.
AI agents assisted with research, implementation, review, and drafting. Nick
Shanin directed the work and takes responsibility for the above text.
was:
AI text below.
h2. Symptom
In a distributed search sorted on a field (so sort values need unmarshalling at
merge time), when one shard answers with partial results and no {{sort_values}}
(the {{timeAllowed}} / query-limit case that SOLR-14758 fixed for an NPE and
SOLR-15595 narrowed), the merging node drops that shard's documents but still
counts the shard in the merged totals. The response shows:
 * {{numFound}} larger than the number of documents reachable by paging,
because it includes the skipped shard's hits
 * that same count reported with {{{}numFoundExact=true{}}}: the skipped
shard's own count is exact, so nothing about the flag is wrong on the shard,
but the total it qualifies counts a shard that contributed no documents
 * a {{maxScore}} that no returned document attains, when the skipped shard
held the highest score
 * an {{approximateTotalHits}} that includes the skipped shard's approximate
count, when the total is written to the response header (11 instead of 4 in the
reproduction below)
The response header does carry {{{}partialResults=true{}}}, so callers know the
result set is partial; nothing tells them the count is not the count of the
merged result.
A second defect sits in the same accumulation. When a shard response carries no
{{approximateTotalHits}} header, the merge falls back to adding
{{{}numFound{}}}, but at that point {{numFound}} is the running total of the
shards merged so far, not the current shard's count, which is only folded in
afterwards. Two shards with 2 and 3 documents and no headers merge to an
approximate total of 2 instead of 5, in arrival order A then B (A the shard
with 2 documents, B the shard with 3; in the other order the total is 3).
h2. Mechanism
{{QueryComponent.mergeIds}} (main,
{{{}solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java{}}})
folds each shard response into the merged totals before the partial-results
skip:
 * {{approximateTotalHits}} is accumulated around line 1160
 * when a shard sends no {{approximateTotalHits}} header, the fallback at line
1162 adds {{{}numFound{}}}, the running merged total at that point, instead of
the current shard's own count, which is only folded in at line 1171
 * {{maxScore}} is merged around line 1169
 * {{numFound += docs.getNumFound()}} at line 1171
 * the {{hitCountIsExact}} fold is at line 1174
The skip itself is at line 1193: {{{}if (thisResponseIsPartial &&
sortFieldValues.size() == 0 && needsUnmarshalling) continue;{}}}. The only
downward adjustment, {{numFound--}} at line 1225, runs per document when the
queue push returns false, which happens only for a document id already pushed
(from any shard), so it never compensates for a shard that was skipped
wholesale. {{setResultIdsAndResponseDocs}} then writes the accumulated values
into the response, including {{{}setNumFoundExact(hitCountIsExact){}}}.
The skipped shard does not rescue the exactness flag: in the SOLR-14758
scenario the timeout lands in {{doFieldSortValues}} after the shard's search
has completed, so the shard's own count is exact and it sends
{{{}numFoundExact=true{}}}; more generally the shard's exact flag comes from
the DocList hit-count relation, which the partial-results state does not
downgrade. So {{hitCountIsExact}} survives the merge. The extreme case is every
shard skipped: on main the merged count is then the skipped shards' sum (5 for
shards reporting 2 and 3) and is reported exact. After the fix the count is
zero, and that zero must not be reported exact when a skipped shard reported
hits.
The response flags are not totals and are not part of this defect:
{{partialResults}} and the max-hits flag still fold in from every shard,
including a skipped one. {{[shards.info|http://shards.info/]}} also still
reports each shard's own totals, so its per-shard numbers do not sum to the
merged totals.
h2. Reproduction
At the merge level, using the mock harness from
{{QueryComponentPartialResultsTest}} (added by SOLR-15595) with realistic
totals set on the fabricated shard lists (that test never sets
{{{}numFound{}}}, which is why this was not visible): shard1 partial with no
{{{}sort_values{}}}, 2 documents, {{{}numFound=2{}}},
{{{}numFoundExact=true{}}}, {{{}maxScore=99.0{}}}, an {{approximateTotalHits}}
header of 7 and {{{}maxHitsTerminatedEarly=true{}}}; shard2 normal, 3
documents, {{{}numFound=3{}}}, {{{}numFoundExact=true{}}},
{{{}maxScore=3.0{}}}, an {{approximateTotalHits}} header of 4; sort on a field
requiring unmarshalling. On main the merged list holds 3 documents (correct)
with {{{}numFound=5{}}}, {{{}numFoundExact=true{}}}, {{maxScore=99.0}} and
{{approximateTotalHits=11}} (4 with the skipped shard excluded).
For the fallback: two shards with 2 and 3 documents, neither sending an
{{approximateTotalHits}} header, one flagging {{{}maxHitsTerminatedEarly{}}};
on main the merged {{approximateTotalHits}} is 2 instead of 5 in arrival order
A then B (3 in the other order).
End to end, the ingredients are the SOLR-14758 ones: a multi-shard collection,
a sorted query, and a {{timeAllowed}} (or another query limit) that trips on
one shard while its sort values are being derived.
h2. Fix options
Three smallest changes present themselves:
 * Test the skip condition before accumulating the shard's totals
(equivalently, move the {{approximateTotalHits}} / {{maxScore}} / {{numFound}}
/ {{hitCountIsExact}} accumulation below the skip), so a skipped shard never
enters the merged totals and the totals describe exactly the shards whose
documents are merged. In the extreme case where every shard is skipped, the
merged count is zero, and that zero must not be reported exact when a skipped
shard reported hits. The reorder alone does not give that: when no shard is
merged, no shard folds its flag either, so the flag keeps its initial true.
That case needs a guard on top, the second option's move applied to the
no-shard-merged case only: clear the flag when no shard was merged and a
skipped shard reported hits.
 * Keep the totals, but set {{hitCountIsExact = false}} when a shard is
skipped, so the response stops asserting the count is exact. That leaves
{{numFound}} and {{maxScore}} counting the skipped shard, so it corrects only
one of the symptoms above.
 * Correct the fallback: when a shard sends no {{approximateTotalHits}} header,
add that shard's own count ({{{}docs.getNumFound(){}}}) instead of the running
merged total.
The first change is the recommended one. Correcting the totals when the skip
fires instead means subtracting the skipped shard's contribution after the
fact. That can repair the counts, but it cannot correctly undo
{{{}maxScore{}}}: a maximum cannot be decremented, so recovering the merged
maximum without the skipped shard would need per-shard tracking and a
recomputation, and {{hitCountIsExact}} would need recomputation as well.
Testing the skip before accumulating leaves every total describing the same set
of shards, so the reorder is the sound fix. The third change is independent of
the skip and is part of the same fix: it is one line in the same accumulation,
and without it the merged approximate total stays wrong whenever a shard sends
no header.
Related: SOLR-14758, SOLR-15595. SOLR-15144 reports a different symptom in the
same area (empty document list under {{{}timeAllowed{}}}) and does not appear
to be the same mechanism. {{CombinedQueryComponent.mergeIds}} folds its
per-query totals in before the same partial skip in the same way. Its
{{approximateTotalHits}} fallback also adds the running per-query total, which
already includes the current shard, so two shards of 2 and 3 documents
accumulate 2 then 7 instead of 5 in arrival order A then B (the other order
accumulates 3 then 8). Its exactness fold is inverted as well: it keeps the
merged flag true only while shards report their counts as not exact, so an
all-exact merge reports inexact and an all-inexact merge reports exact. That
component is not covered here and is left as a follow-up, as one ticket
covering all three of its defects.
AI agents assisted with research, implementation, review, and drafting. Nick
Shanin directed the work and takes responsibility for the above text.
> QueryComponent.mergeIds counts a partial shard whose documents are skipped in
> the merged totals, and adds the running total as the approximateTotalHits
> fallback
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18539
> URL: https://issues.apache.org/jira/browse/SOLR-18539
> Project: Solr
> Issue Type: Bug
> Components: query
> Reporter: Nick Shanin
> Priority: Major
>
> 🤖 _AI text below_ 🤖 _(posted on behalf of Nick Shanin)_
> In a distributed search sorted on a field, {{QueryComponent.mergeIds}} folds
> each shard's totals into the merged response before it tests the
> partial-results skip, so a shard whose documents are dropped as partial still
> counts. This was confirmed by a run on unmodified main with fabricated shard
> responses: the merged list held 3 documents, but {{numFound}} was 5,
> {{numFoundExact}} was true, {{maxScore}} was 99.0 from the skipped shard, and
> {{approximateTotalHits}} was 11 instead of 4. A second defect sits in the
> same code: when a shard sends no {{approximateTotalHits}} header, the
> fallback adds the running merged total instead of that shard's own count, so
> two shards with 2 and 3 documents merged to an approximate total of 2 instead
> of 5.
> The fix I recommend is to test the skip before accumulating, so a skipped
> shard never enters the totals; subtracting its contribution afterwards cannot
> work, because a maximum such as {{maxScore}} cannot be decremented and the
> exactness flag would need recomputation. Setting {{hitCountIsExact}} to false
> when a shard is skipped would fix only the exactness symptom and leave the
> counts wrong. The fallback should also add the shard's own count instead of
> the running total, as part of the same fix.
> AI agents assisted with research, implementation, review, and drafting. Nick
> Shanin directed the work and takes responsibility for the above text.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
