[
https://issues.apache.org/jira/browse/LUCENE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917158#comment-13917158
]
Gilad Barkai commented on LUCENE-5476:
--------------------------------------
Great effort!
I wish to through in another part - the description of this issue is about
sampling, but the implementation is about *random* sampling.
This is not always the case, nor it is very fast (indeed, calling 1M times
Random.nextInt would be measurable by itself IMHO).
A different sample could be
{{code}}
int acceptedModulu = (int)(1/sampleRatio);
int next() {
do {
nextDoc = inner.next();
} while (nextDoc != NO_MORE_DOCX && nextDoc % acceptedModulu != 0) ;
return nextDoc;
}
{{code}}
This should be faster as a sampler, and perhaps saves us from creating a new
{DocIdSet}.
One last thing - if I did the math right - the sample crafted by the code in
the patch would be twice as large as the user may expect.
For a sample ratio of 0.1, the random.nextInt() would be called with 10, so the
avg. "jump" is actually 5 - and every 5th document in the original set (again,
in avg) would be selected, and not every 10th in avg. I think the
random.nextInt should be called with twice the size it is called now (e.g 20,
making the avg random selection 10).
> Facet sampling
> --------------
>
> Key: LUCENE-5476
> URL: https://issues.apache.org/jira/browse/LUCENE-5476
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Rob Audenaerde
> Attachments: LUCENE-5476.patch, SamplingFacetsCollector.java
>
>
> With LUCENE-5339 facet sampling disappeared.
> When trying to display facet counts on large datasets (>10M documents)
> counting facets is rather expensive, as all the hits are collected and
> processed.
> Sampling greatly reduced this and thus provided a nice speedup. Could it be
> brought back?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
