[
https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408097#comment-13408097
]
Mark Harwood commented on LUCENE-4069:
--------------------------------------
Thanks for the extra tests, Mike. That's tightened performance but that lookss
a scary amount of code for the optimal solution of this basic incrementing
operation :)
I've done some more benchmarks with the updated test and the performance
characteristics are becoming clearer as shown in these results:
http://goo.gl/dtWSb
Bloom performance is better than Pulsing but the gap narrows with the volumes
of deletes lying around in old segments, caused by updates. In these cases the
BloomFilter gives a false positive and falls back to the equivalent operations
of Pulsing. I added a 100mb start size for the BloomFilter for large-scale
tests because without this it gets saturated and there were occasional big
spikes in batch times.
So overall there still looks to be a benefit and especially in low-frequency
update scenarios.
I'll wait for the dust to settle on Lucene-4190 (given this Codec introduces a
new file) before thinking about committing.
Cheers
Mark
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
> Key: LUCENE-4069
> URL: https://issues.apache.org/jira/browse/LUCENE-4069
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 3.6, 4.0
> Reporter: Mark Harwood
> Priority: Minor
> Fix For: 4.0, 3.6.1
>
> Attachments: BloomFilterPostingsBranch4x.patch,
> LUCENE-4069-tryDeleteDocument.patch, MHBloomFilterOn3.6Branch.patch,
> PKLookupUpdatePerfTest.java, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PrimaryKeyPerfTest40.java
>
>
> An addition to each segment which stores a Bloom filter for selected fields
> in order to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with
> many segments but also speeds up general searching in my tests.
> Overview slideshow here:
> http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm"
> on the end of the name to invoke special indexing/querying capability.
> Clearly a new Field or schema declaration(!) would need adding to APIs to
> configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
