[
https://issues.apache.org/jira/browse/LUCENE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264201#comment-15264201
]
Adrien Grand commented on LUCENE-7258:
--------------------------------------
I played with the scaling factor and the "poly 5" geo benchmark by temporarily
switching from MatchingPoints to DocIdSetBuilder in LatLonPointInPolygonQuery.
I got the following QPS:
||scaling factor = 9/8 (like in master)|48.3|
||scaling factor = 5/4|49.3|
||scaling factor = 3/2|50.2|
||scaling factor = 2|50.9|
||MatchingPoints|51.7|
This gets DocIdSetBuilder closer to the throughput of MatchingPoints in spite
of the fact it tries to better deal with the sparse case. Given than wasting
space is not a big deal for this class (the data will be trashed once the query
finishes running), I would be in favor of moving to a scaling factor of 3/2 or
2.
Regarding reusing fixed bitsets, I think the only way would be to keep state on
the index searcher and then have access to the cache in {{Query.createWeight}}.
But I don't think I would like it: this looks quite dangerous to me as bit sets
can take a lot of memory and you need a different cache per thread (if your
index has 1B documents, you would need 120MB per thread for a single
FixedBitSet, while a single query may need to create several of them).
> Tune DocIdSetBuilder allocation rate
> ------------------------------------
>
> Key: LUCENE-7258
> URL: https://issues.apache.org/jira/browse/LUCENE-7258
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial
> Reporter: Jeff Wartes
> Attachments:
> LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch,
> LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch,
> allocation_plot.jpg
>
>
> LUCENE-7211 converted IntersectsPrefixTreeQuery to use DocIdSetBuilder, but
> didn't actually reduce garbage generation for my Solr index.
> Since something like 40% of my garbage (by space) is now attributed to
> DocIdSetBuilder.growBuffer, I charted a few different allocation strategies
> to see if I could tune things more.
> See here: http://i.imgur.com/7sXLAYv.jpg
> The jump-then-flatline at the right would be where DocIdSetBuilder gives up
> and allocates a FixedBitSet for a 100M-doc index. (The 1M-doc index
> curve/cutoff looked similar)
> Perhaps unsurprisingly, the 1/8th growth factor in ArrayUtil.oversize is
> terrible from an allocation standpoint if you're doing a lot of expansions,
> and is especially terrible when used to build a short-lived data structure
> like this one.
> By the time it goes with the FBS, it's allocated around twice as much memory
> for the buffer as it would have needed for just the FBS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
