[
https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286916#comment-13286916
]
Mark Harwood commented on LUCENE-4069:
--------------------------------------
bq. why is this a speed improvement?
Sorry - misleading. Replace the word "faster" in my comment with "better" and
that makes more sense - I mean better in terms of resource usage and reduced
open file handles. This seemed relevant given the earlier comments about Solr's
use of non-compound files:
bq. [Solr] create massive amounts of files if we did so (add to the fact it
disables compound files by default and its a disaster...)
I can see there is a useful simplification being sought for here if PerFieldPF
can consider each of the unique top-level PFs presented to it as looking after
an exclusive set of files. As the centralised allocator of file names it can
then simply call each unique PF with a choice of segment suffix to name its
various files without conflicting with other PFs. Lucene 4093 is all about
better determining which PF is unique using .equals(). Unfortunately I don't
think this approach is sufficiently complex. In order to avoid allocating all
unnecessary file names PerFieldPF would have to further understand the nuances
of which PFs were being wrapped by other PFs and which wrapped PFs would be
reusable outside of their wrapped PF (as is the case with BloomPF's wrapped
PF). That seems a more complex task than implementing equals().
So it seems we have 3 options:
1) Ignore the problems of creating too many files in the case of BloomPF and
any other examples of "wrapping" PFs
2) Create a PerFieldPF implementation that reuses wrapped PFs using some
generic means of discovering recyclable wrapped PFs (i.e go further than what
4093 currently proposes in adding .equals support)
3) Retain my BloomPF-specific solution to the problem for those prepared to use
lower-level APIs.
Am I missing any other options and which one do you want to go for?
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
> Key: LUCENE-4069
> URL: https://issues.apache.org/jira/browse/LUCENE-4069
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 3.6, 4.0
> Reporter: Mark Harwood
> Priority: Minor
> Fix For: 4.0, 3.6.1
>
> Attachments: BloomFilterPostings40.patch,
> MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields
> in order to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with
> many segments but also speeds up general searching in my tests.
> Overview slideshow here:
> http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm"
> on the end of the name to invoke special indexing/querying capability.
> Clearly a new Field or schema declaration(!) would need adding to APIs to
> configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
