[
https://issues.apache.org/jira/browse/SOLR-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672570#comment-16672570
]
Hoss Man commented on SOLR-12954:
---------------------------------
The code that generates shard requests has an optimization for dealing with the
per-shard mincount...
{code:java}
// per-shard mincount & overrequest
if ( FacetParams.FACET_SORT_INDEX.equals(sort) &&
1 < requestedMinCount &&
0 < requestedLimit) {
// We can divide the mincount by num shards rounded up, because unless
// a single shard has at least that many it can't compete...
shardMinCount = (int) Math.ceil((double) requestedMinCount /
rb.slices.length);
{code}
...but this optimization depends on *all* candidate terms being refined (when
using sort=index) – there is evidently some situation where this is not
happening properly.
----
With {{cf445ba54998710466a7c6cb489d3162d20d127a}} checked out, the following
seed demonstrates the problem...
{noformat}
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestCloudPivotFacet
-Dtests.method=test -Dtests.seed=8A3362235855F8A8 -Dtests.multiplier=2
-Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=sr-Latn -Dtests.timezone=America/Guatemala -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 38.9s | TestCloudPivotFacet.test <<<
[junit4] > Throwable #1: java.lang.AssertionError:
{main(facet=true&facet.pivot={!stats%3Dst3}pivot_dt,pivot_b1&facet.pivot={!stats%3Dst2}pivot_z_s1,pivot_i1,pivot_d1&facet.limit=12&facet.pivot.mincount=4&facet.sort=index&facet.overrequest.count=2&facet.overrequest.ratio=1.1145182),extra(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index)}
==> pivot_dt,pivot_b1:
{params(rows=0),defaults({main({main(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index),extra(fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z)}),extra(fq={!term+f%3Dpivot_b1}false)})}
expected:<5> but was:<6>
[junit4] > at
__randomizedtesting.SeedInfo.seed([8A3362235855F8A8:2675DF9F6A99550]:0)
[junit4] > at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:292)
[junit4] > at
org.apache.solr.cloud.TestCloudPivotFacet.test(TestCloudPivotFacet.java:239)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > Caused by: java.lang.AssertionError: pivot_dt,pivot_b1:
{params(rows=0),defaults({main({main(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index),extra(fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z)}),extra(fq={!term+f%3Dpivot_b1}false)})}
expected:<5> but was:<6>
[junit4] > at
org.apache.solr.cloud.TestCloudPivotFacet.assertNumFound(TestCloudPivotFacet.java:690)
[junit4] > at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotData(TestCloudPivotFacet.java:346)
[junit4] > at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:313)
[junit4] > at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:324)
[junit4] > at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:282)
[junit4] > ... 42 more
{noformat}
The pivot query being tests...
{noformat}
[junit4] 2> 35268 INFO
(TEST-TestCloudPivotFacet.test-seed#[8A3362235855F8A8]) [ ]
o.a.s.c.TestCloudPivotFacet Doing full run:
{main(facet=true&facet.pivot={!stats%3Dst3}pivot_dt,pivot_b1&facet.pivot={!stats%3Dst2}pivot_z_s1,pivot_i1,pivot_d1&facet.limit=12&facet.pivot.mincount=4&facet.sort=index&facet.overrequest.count=2&facet.overrequest.ratio=1.1145182),extra(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index)}
{noformat}
The bucket that fails is from the {{pivot_dt,pivot_b1}} pivot for the bucket
{{2010-10-31T10:31:15Z , false}}
The test verification queries show...
{noformat}
### verification queries of 2010-10-31T10:31:15Z (shard1=2 shard2=7)...
[junit4] 2> 36749 INFO (qtp1638186575-84) [n:127.0.0.1:51294__
c:collection1 s:shard2 r:core_node42 x:collection1_shard2_replica_n41]
o.a.s.c.S.Request [collection1_shard2_replica_n41] webapp=/_ path=/select
params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&_test_min=4&shard.url=http://127.0.0.1:51294/_/collection1_shard2_replica_n41/|http://127.0.0.1:59609/_/collection1_shard2_replica_n45/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572502&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1}
hits=7 status=0 QTime=1
[junit4] 2> 36749 INFO (qtp1007717333-122) [n:127.0.0.1:55141__
c:collection1 s:shard1 r:core_node44 x:collection1_shard1_replica_n43]
o.a.s.c.S.Request [collection1_shard1_replica_n43] webapp=/_ path=/select
params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&_test_min=4&shard.url=http://127.0.0.1:55141/_/collection1_shard1_replica_n43/|http://127.0.0.1:38554/_/collection1_shard1_replica_n47/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572502&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1}
hits=2 status=0 QTime=1
[junit4] 2> 36750 INFO (qtp73311073-158) [n:127.0.0.1:59609__
c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45]
o.a.s.c.S.Request [collection1_shard2_replica_n45] webapp=/_ path=/select
params={q=id:[*+TO+1077]&stats=true&_test_sort=index&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&_test_min=4&rows=0&wt=javabin&version=2&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1}
hits=9 status=0 QTime=3
### verification queries of 2010-10-31T10:31:15Z + false (shard1=1, shard2=5)...
[junit4] 2> 36754 INFO (qtp1638186575-84) [n:127.0.0.1:51294__
c:collection1 s:shard2 r:core_node42 x:collection1_shard2_replica_n41]
o.a.s.c.S.Request [collection1_shard2_replica_n41] webapp=/_ path=/select
params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&fq={!term+f%3Dpivot_b1}false&_test_min=4&shard.url=http://127.0.0.1:51294/_/collection1_shard2_replica_n41/|http://127.0.0.1:59609/_/collection1_shard2_replica_n45/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572507&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1}
hits=5 status=0 QTime=1
[junit4] 2> 36756 INFO (qtp1007717333-124) [n:127.0.0.1:55141__
c:collection1 s:shard1 r:core_node44 x:collection1_shard1_replica_n43]
o.a.s.c.S.Request [collection1_shard1_replica_n43] webapp=/_ path=/select
params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&fq={!term+f%3Dpivot_b1}false&_test_min=4&shard.url=http://127.0.0.1:55141/_/collection1_shard1_replica_n43/|http://127.0.0.1:38554/_/collection1_shard1_replica_n47/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572507&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1}
hits=1 status=0 QTime=1
[junit4] 2> 36758 INFO (qtp973539008-202) [n:127.0.0.1:38554__
c:collection1 s:shard1 r:core_node48 x:collection1_shard1_replica_n47]
o.a.s.c.S.Request [collection1_shard1_replica_n47] webapp=/_ path=/select
params={q=id:[*+TO+1077]&stats=true&_test_sort=index&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&fq={!term+f%3Dpivot_b1}false&_test_min=4&rows=0&wt=javabin&version=2&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1}
hits=6 status=0 QTime=6
{noformat}
Based on the computed shard min count used, and the per-shard numFounds for the
verification queries, this bucket would not have been returned by shard1 during
the initial query, but would have been returned by shard2 – yet there is
nothing in the logs showing any attempt at refining it on shard2, and the
resulting assertion failure is consistent with the one doc in this bucket on
shard1 not being counted.
----
Forcing the above mentioned code to use {{shardMinCount=1}} causes this seed to
pass, and that may be the best course of action if we can't figure out why the
buckets aren't getting refined properly.
> facet.pivot refinement bug when facet.sort=index and mincount > 2*numShards
> ---------------------------------------------------------------------------
>
> Key: SOLR-12954
> URL: https://issues.apache.org/jira/browse/SOLR-12954
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Hoss Man
> Priority: Major
>
> While testing out SOLR-7804 i discovered a failure in TestCloudPivotFacet
> that indicates a problem with the refinement of (nested?) pivot facets when
> {{facet.sort=index}} and {{facet.pivot.mincount > 2*numShards}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
