[
https://issues.apache.org/jira/browse/SOLR-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053208#comment-14053208
]
Steve Molloy commented on SOLR-3583:
------------------------------------
Tried applying patch with latest SOLR-2894 patch (2014-07-02) on top of 4.9 and
get a failure in the unit test. Already switched the commented lines around
dates (using parsed dates instead of strings) and that helps. Bute no it seems
to not be getting same results on single shard and distributed. I get the
following output when running ant test for it:
[junit4] 2> 7009 T57 C8 oasc.SolrCore.execute [collection1] webapp=
path=/select
params={f.hourlypay_d.percentiles.gap=1.0&facet.field=hiredate_dt&percentiles=true&f.hourlypay_d.percentiles_buckets=true&f.hourlypay_d.percentiles.upper.fence=150.0&f.yearlypay_ti.percentiles_buckets=true&facet.pivot=company_s,hiredate_dt&percentiles.requested.percentiles=25,50,75,80,91,96,97,99&f.yearlypay_ti.percentiles.lower.fence=5000&f.yearlypay_ti.percentiles.gap=1000&f.hiredate_dt.percentiles.upper.fence=2012-07-30T00:00:00Z&f.hourlypay_d.percentiles.lower.fence=5.50&rows=0&percentiles.field=yearlypay_ti&percentiles.field=hourlypay_d&percentiles.field=hiredate_dt&version=2&q=*:*&shards=[ff01::114]:33332/|[ff01::213]:33332/|https://127.0.0.1:57329/|[ff01::083]:33332/,https://127.0.0.1:36083/&f.yearlypay_ti.percentiles.upper.fence=500000&f.hiredate_dt.percentiles.gap=%2B1DAYS&f.hiredate_dt.percentiles.lower.fence=2012-05-01T00:00:00Z&f.hiredate_dt.percentiles_buckets=true&facet=true&wt=javabin}
hits=10 status=0 QTime=44
[junit4] 2> 7019 T12
oas.BaseDistributedSearchTestCase.compareSolrResponses ERROR Mismatched
responses:
[junit4] 2>
{responseHeader={status=0,QTime=44},response={numFound=10,start=0,maxScore=1.0,docs=[]},facet_counts={facet_queries={},facet_fields={hiredate_dt={2012-07-01T12:30:00Z=10}},facet_dates={},facet_ranges={},facet_pivot={company_s,hiredate_dt=[{field=company_s,value=Stark
Industries,count=10,statistics={yearlypay_ti={percentiles={25.0=31500,50.0=35500,75.0=37500,80.0=37500,91.0=37500,96.0=37500,97.0=37500,99.0=37500}},hourlypay_d={percentiles={25.0=91.0,50.0=95.0,75.0=95.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}},pivot=[{field=hiredate_dt,value=Sun
Jul 01 08:30:00 EDT
2012,count=10,statistics={yearlypay_ti={percentiles={25.0=31500,50.0=35500,75.0=37500,80.0=37500,91.0=37500,96.0=37500,97.0=37500,99.0=37500}},hourlypay_d={percentiles={25.0=91.0,50.0=95.0,75.0=95.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}}}]}]}}}
[junit4] 2>
{responseHeader={status=0,QTime=6},response={numFound=10,start=0,docs=[]},facet_counts={facet_queries={},facet_fields={hiredate_dt={2012-07-01T12:30:00Z=10}},facet_dates={},facet_ranges={},facet_pivot={company_s,hiredate_dt=[{field=company_s,value=Stark
Industries,count=10,statistics={yearlypay_ti={percentiles={25.0=32500,50.0=35500,75.0=37500,80.0=37500,91.0=38500,96.0=38500,97.0=38500,99.0=38500}},hourlypay_d={percentiles={25.0=95.0,50.0=95.0,75.0=100.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}},pivot=[{field=hiredate_dt,value=Sun
Jul 01 08:30:00 EDT
2012,count=10,statistics={yearlypay_ti={percentiles={25.0=32500,50.0=35500,75.0=37500,80.0=37500,91.0=38500,96.0=38500,97.0=38500,99.0=38500}},hourlypay_d={percentiles={25.0=95.0,50.0=95.0,75.0=100.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}}}]}]}}}
[junit4] 2> 7044 T12 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=1953253883
[junit4] 2> 7045 T12 oasc.SolrCore.close [collection1] CLOSING SolrCore
org.apache.solr.core.SolrCore@25dc8c85
[junit4] 2> 7045 T12 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=4,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=40,cumulative_deletesById=0,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=4755,transaction_logs_total_number=4}
[junit4] 2> 7046 T12 oasu.SolrCoreState.decrefSolrCoreState Closing
SolrCoreState
[junit4] 2> 7046 T12 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 7046 T12 oasu.DefaultSolrCoreState.closeIndexWriter closing
IndexWriter with IndexWriterCloser
[junit4] 2> 7047 T12 oasc.SolrCore.closeSearcher [collection1] Closing
main searcher on request.
[junit4] 2> 7047 T12 oasc.CachingDirectoryFactory.close Closing
MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.closeCacheValue looking
to close
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data/index
[CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data/index;done=false>>]
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.close Closing directory:
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data/index
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.closeCacheValue looking
to close
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data
[CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data;done=false>>]
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.close Closing directory:
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data
[junit4] 2> 7049 T12 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 7123 T12 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=1841909798
[junit4] 2> 7123 T12 oasc.SolrCore.close [collection1] CLOSING SolrCore
org.apache.solr.core.SolrCore@35e21988
[junit4] 2> 7124 T12 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=4,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=22,cumulative_deletesById=0,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=2719,transaction_logs_total_number=4}
[junit4] 2> 7124 T12 oasu.SolrCoreState.decrefSolrCoreState Closing
SolrCoreState
[junit4] 2> 7124 T12 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 7124 T12 oasu.DefaultSolrCoreState.closeIndexWriter closing
IndexWriter with IndexWriterCloser
[junit4] 2> 7125 T12 oasc.SolrCore.closeSearcher [collection1] Closing
main searcher on request.
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.close Closing
MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.closeCacheValue looking
to close
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data
[CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data;done=false>>]
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.close Closing directory:
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.closeCacheValue looking
to close
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data/index
[CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data/index;done=false>>]
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.close Closing directory:
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data/index
[junit4] 2> 7127 T12 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 7199 T12 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=1421481657
[junit4] 2> 7200 T12 oasc.SolrCore.close [collection1] CLOSING SolrCore
org.apache.solr.core.SolrCore@2e4c44b2
[junit4] 2> 7200 T12 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=4,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=18,cumulative_deletesById=0,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=2376,transaction_logs_total_number=4}
[junit4] 2> 7205 T12 oasu.SolrCoreState.decrefSolrCoreState Closing
SolrCoreState
[junit4] 2> 7205 T12 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 7205 T12 oasu.DefaultSolrCoreState.closeIndexWriter closing
IndexWriter with IndexWriterCloser
[junit4] 2> 7206 T12 oasc.SolrCore.closeSearcher [collection1] Closing
main searcher on request.
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.close Closing
MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.closeCacheValue looking
to close
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data/index
[CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data/index;done=false>>]
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.close Closing directory:
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data/index
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.closeCacheValue looking
to close
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data
[CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data;done=false>>]
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.close Closing directory:
./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data
[junit4] 2> 7208 T12 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 7259 T12 oas.SolrTestCaseJ4.tearDown ###Ending
testDistribSearch
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=DistributedFacetPivotStatisticsTest -Dtests.method=testDistribSearch
-Dtests.seed=CD32BB6D12588826 -Dtests.slow=true -Dtests.locale=ms
-Dtests.timezone=America/Indiana/Indianapolis -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 6.82s |
DistributedFacetPivotStatisticsTest.testDistribSearch <<<
[junit4] > Throwable #1: junit.framework.AssertionFailedError:
.facet_counts.facet_pivot.company_s,hiredate_dt[0].statistics.yearlypay_ti.percentiles.25.0:31500!=32500
[junit4] > at
__randomizedtesting.SeedInfo.seed([CD32BB6D12588826:4CD435756507E81A]:0)
[junit4] > at junit.framework.Assert.fail(Assert.java:50)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:838)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:857)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:560)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:540)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:519)
[junit4] > at
org.apache.solr.handler.component.DistributedFacetPivotStatisticsTest.doTest(DistributedFacetPivotStatisticsTest.java:430)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:871)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 7282 T12 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at:
/home/smolloy/workspace-solr/otif-trunk/solr/build/solr-core/test/J0/./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001
[junit4] 2> NOTE: test params are: codec=Lucene49:
{hiredate_dt=Pulsing41(freqCutoff=19 minBlockSize=100 maxBlockSize=291),
_version_=FSTOrd41, multiDefault=FST41, yearlypay_ti=FSTOrd41,
intDefault=FSTOrd41, range_facet_sl=Pulsing41(freqCutoff=19 minBlockSize=100
maxBlockSize=291), range_facet_si=FST41,
company_s=PostingsFormat(name=Lucene41WithOrds),
id=PostingsFormat(name=Lucene41WithOrds), hourlypay_d=FST41,
range_facet_l=FSTOrd41, timestamp=FSTOrd41}, docValues:{},
sim=DefaultSimilarity, locale=ms, timezone=America/Indiana/Indianapolis
[junit4] 2> NOTE: Linux 3.13.0-30-generic amd64/Oracle Corporation
1.8.0_05 (64-bit)/cpus=8,threads=1,free=170392096,total=206045184
[junit4] 2> NOTE: All tests run in this JVM:
[DistributedFacetPivotStatisticsTest]
[junit4] Completed in 7.38s, 1 test, 1 failure <<< FAILURES!
[junit4]
[junit4] JVM J0: stderr was not empty, see:
/home/smolloy/workspace-solr/otif-trunk/solr/build/solr-core/test/temp/junit4-J0-20140706_153759_130.syserr
[junit4] >>> JVM J0: stderr (verbatim) ----
[junit4] Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=128m; support was removed in 8.0
[junit4] <<< JVM J0: EOF ----
[junit4]
[junit4] Tests with failures:
[junit4] -
org.apache.solr.handler.component.DistributedFacetPivotStatisticsTest.testDistribSearch
[junit4]
[junit4]
[junit4] JVM J0: 0.62 .. 8.67 = 8.06s
[junit4] Execution time total: 8.68 sec.
[junit4] Tests summary: 1 suite, 1 test, 1 failure
Seems some values are not considered in distributed response. Do you have
pointers where to look for cause of differences?
> Percentiles for facets, pivot facets, and distributed pivot facets
> ------------------------------------------------------------------
>
> Key: SOLR-3583
> URL: https://issues.apache.org/jira/browse/SOLR-3583
> Project: Solr
> Issue Type: Improvement
> Reporter: Chris Russell
> Priority: Minor
> Labels: newbie, patch
> Fix For: 4.9, 5.0
>
> Attachments: SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch,
> SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch
>
>
> Built on top of SOLR-2894, this patch adds percentiles and averages to
> facets, pivot facets, and distributed pivot facets by making use of range
> facet internals.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
