[
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464131#comment-16464131
]
Erick Erickson commented on LUCENE-7976:
----------------------------------------
Simon:
Thanks for taking the time, and yeah, the complexity bothers me too.
I'm reluctant to take the comments out for 8263 as their lifetime is expected
to be a few days after this is checked in and I don't want to have to remember
how I got it to work, but we can if you strongly object.
About tests. There are actually a number of tests for TMP functionality
already, in particular merging down to N segments and forcemerging, what
particular parts of the code do you think should be exercised more?
As for naming/style changes, sure. I always feel like an interloper in the
Lucene code; I'm perfectly willing to try to keep it consistent with the
folks' expectations who, you know live in the Lucene codebase ;)
As for your other suggestions, I don't know. I've rewritten/refactored/whatever
this about three times already and every time I do it gets complicated again.
bq. I wonder if we can generalize the algorithm here into a single method
because in the end they all do the same thing.
doFindMerges is called from all three operations so I'm a bit puzzled about
what this would look like. Each of the three operations has different initial
conditions, eligible segments, max segment size allowed and the like. Once
those are established, they all go through the same scoring/selection code.
Figuring out the initial conditions is "interesting"
There are several areas here that are particularly gnarly and any way to
un-gnarl them would be great.
> the variable number of segments in findForcedMerges. I'd love for there to be
> exactly two choices; merge down to one or respect max segment size.
> Controlling the number of segments would then be more along the lines of
> setting maxMergedSegmentsMB in TMP.
> the multi-pass nature of findForcedMerges because it respects the
> maxMergeAtOnceExplicit.
> Somewhat different decisions need to be made in doFindMerges depending on
> what type of merge it is. I'm not a huge fan of passing that enum in. One of
> the iterations tried to pass information into an uber-function but that lead
> to having to pass around segmentsToMerge from findForcedMerges, which wasn't
> present in the other two, so passing null in from them was also ugly.
> I also ran into a couple of issues with findMerges needing to _not_ merge
> segments if there weren't enough in the tier, which is exactly the opposite
> of findForcedMerges, which itself has conditions around whether it should
> merge a segment with no deletions or not if exceeds maxMergedSegmentMB which
> itself is a variable condition based on whether maxSegments has been
> specified.....
Let me know if you're going to take a whack at it, even a skeleton of a
different approach would help me get on the same page.
Meanwhile I can incorporate your other comments, they'll be useful no matter
what.
> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of
> very large segments
> -------------------------------------------------------------------------------------------------
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Major
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch,
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on
> disk) handled quite easily in a single Lucene index. This is particularly
> true as features like docValues move data into MMapDirectory space. The
> current TMP algorithm allows on the order of 50% deleted documents as per a
> dev list conversation with Mike McCandless (and his blog here:
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many
> TB) solutions like "you need to distribute your collection over more shards"
> become very costly. Additionally, the tempting "optimize" button exacerbates
> the issue since once you form, say, a 100G segment (by
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like
> <maxAllowedPctDeletedInBigSegments> (no, that's not serious name, suggestions
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with
> >> smaller segments to bring the resulting segment up to 5G. If no smaller
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize).
> >> It would be rewritten into a single segment removing all deleted docs no
> >> matter how big it is to start. The 100G example above would be rewritten
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the
> default would be the same behavior we see now. As it stands now, though,
> there's no way to recover from an optimize/forceMerge except to re-index from
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the
> wild" with 10s of shards replicated 3 or more times. And that doesn't even
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A
> new merge policy is certainly an alternative.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
