[
https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415219#comment-13415219
]
Michael McCandless commented on LUCENE-4069:
--------------------------------------------
bq. Estimating key volumes in context 1 is probably hard without some
additional hints from the end user.
Actually, the indexer knows exactly how many unique terms it's about to
flush. It's the unique term count (for this one segment) that you
need right? We just have to plumb it somehow (add to
SegmentWriteState?).
bq. 2) Merged segments e.g. guessing how many unique keys survive a merge
operation
Seems like LUCENE-4198 needs to solve this same problem.
bq. Not sure how we get the OneMerge instance fed through the call stack -
could that be held somewhere on a ThreadLocal as generally useful context?
Yeah I'm not sure either ... but I agree an approx heuristic based on
merging segments unique term counts is better than nothing.
I don't like how you silently just get bad perf now if your up-front
guess was too small ... I also don't like that the default guessing is
static (8 MB): flushing tiny NRT segs are gonna pay an absurd price.
I think we need a solution for this...
Maybe another option is to simply make a 2nd pass after the segment
(flush or merge) is written, to build up the bit set; this way we know
exactly how many unique terms we have.
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
> Key: LUCENE-4069
> URL: https://issues.apache.org/jira/browse/LUCENE-4069
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 3.6, 4.0-ALPHA
> Reporter: Mark Harwood
> Priority: Minor
> Fix For: 4.0
>
> Attachments: BloomFilterPostingsBranch4x.patch,
> LUCENE-4069-tryDeleteDocument.patch, LUCENE-4203.patch,
> MHBloomFilterOn3.6Branch.patch, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PrimaryKeyPerfTest40.java
>
>
> An addition to each segment which stores a Bloom filter for selected fields
> in order to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with
> many segments but also speeds up general searching in my tests.
> Overview slideshow here:
> http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm"
> on the end of the name to invoke special indexing/querying capability.
> Clearly a new Field or schema declaration(!) would need adding to APIs to
> configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
