[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054293#comment-14054293
]
Hoss Man commented on SOLR-2894:
--------------------------------
bq. ... – but as things stand right now there is still a nasty bug somewhere in
the facet.missing processing that i can't wrap my head arround...
I spent today doing some manual testing with some small amounts of data, and
looking at the shard requests triggered by each request. I then started
reading through more of the refinement code (first time i've looked at a fair
bit of this) and i think i've figured out what's going on (but i don't have a
fix for it yet)...
Basically: the {{PivotFacetField}} class, that holds a
{{List<PivotFacetValue>}} doesn't do anything special as far as keeping track
of the PivotFacetValue that represents the {{facet.missing}} value (ie: the
PivotFacetValue where {{PivotFacetValue.value==null}}). This means that in
methods like {{PivotFacetField.sort()}} and
{{PivotFacetField.queuePivotRefinementRequests(...)}} the PivotFacetValue for
{{facet.missing}} is mixed in with the other values and included in
considerations about what the cutoff "countThreshold" is for refinement, _even
though it's not affected by {{facet.limit}} and should always be returned_
This means that in the test i added that has
{{facet.limit=1&facet.missing=true}} the {{null}} vaue from {{facet.missing}}
is the only value considered in the "top 1" of the constraints, and has a count
much higher then the count for the SPECIAL value -- which means SPECIAL doesn't
even qualify for the {{processPossibleCandidateElement}} logic so it never gets
refined at all.
--
I think the best course of action is to cleanup {{PivotFacetField}} a bit, so
that in addition to the {{List<PivotFacetValue>}} of values that are subject to
the facet.limit, a specfic "missingValue" variable should be added to track the
corrisponding {{PivotFacetValue}} -- this should make the value sorting &
refinement logic in {{queuePivotRefinementRequests()}} accurate as is, at the
cost of slightly more complex (but accurately modeled) logic in
{{createFromListOfNamedLists()}} and {{convertToListOfNamedLists()}}.
What do folks think?
----
bq. The endless loop is definitely a concern and we will focus on that first if
your changes haven't already fixed that.
The root cause of the infinite loop seems to be that the formating/parsing of
of the refinment params wasn't in sync (ie: empty strings weren't being
included at all, while facet.missing values were being encoded as "null" which
owuld then be parsed as 4 character string literals) ... so that _cause_ should
be fixed in my latest patch.
What still concerns me though is that there is evidently no general sanity
check in the code to prevent the distributed logic in the coordinator from
retrying to refine values over and over again even if the shard never responses
back with a number for it (ie: if some future bug gets introduced in the refine
code that runs on the shards, or if some shard has been misconfigured to have a
hard coded invaraint of {{facet=false}}, etc...). That's the sort of edge case
that may be really hard to test for, but even if we can't explicit test it, we
should at least have some sanity check in the distrib coordination code that
says "we already asked shardX for refinementY and still don't have it, throw
5xx error!" instead of "still need refinementY from shardX, ..., still need
refinementY from shardX, ..." which is what seems to be happening right now.
bq. What we are wondering is if you feel we could get a preliminary version of
this committed if we can resolve that loop?
I'm not comfortable committing features unless i know they work -- particularly
something like this, where it's adding distributed support to an existing core
feature. I don't want existing pivot users to see "oh, distributed pivot
support has been added" and upgrade to SolrCloud and then start getting
silently incorrect results.
Rest assured however: I'm dedicated to continuing to working through this
issue, and helping to fix whatever bugs we find, until it's ready to be
committed. I won't leave you hanging.
----
bq. Hey Hoss, I should have time this week and next to investigate the infinite
loop and try to implement some of your other requests.
That's great -- like i mentioned above, i think a sanity check on the infinite
loop is important, but i suspect it should be fairly trivial (i'm just not 100%
certain where it makes sense to put it yet)
I think the biggest concern right now however is addressing the bugs with how
{{facet.missing}} impacts refinement and kicks value values out of contention
due to the modeling in {{PivotFacetField}}
If you have time, and can help out with making those changes, I can go ahead
and focus on the additional tests i was describing -- which is probably the
best way to divide & conquer the problem since you guys already know the code
internals better then me. Then you can help review my tests, and i can help
review the {{PivotFacetField}} changes.
sound good?
> Implement distributed pivot faceting
> ------------------------------------
>
> Key: SOLR-2894
> URL: https://issues.apache.org/jira/browse/SOLR-2894
> Project: Solr
> Issue Type: Improvement
> Reporter: Erik Hatcher
> Assignee: Hoss Man
> Fix For: 4.9, 5.0
>
> Attachments: SOLR-2894-mincount-minification.patch,
> SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
> SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh
>
>
> Following up on SOLR-792, pivot faceting currently only supports
> undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
