[
https://issues.apache.org/jira/browse/SOLR-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832496#comment-15832496
]
Michael Sun commented on SOLR-10015:
------------------------------------
bq. If one of those large entries in the cache were to have a percentage of its
elements removed by GC, I think that search results would be incorrect.
It won't create incorrect result. If any entry is pruned by GC, it would be
simply regenerated which should be the identical to the original one.
bq. I think this is only likely to delay OOM instead of preventing it, and even
if it did prevent it, a low memory situation would probably involve constant
eviction (by GC) and recreation (by queries) of FC entries, which I would
expect to have strong performance implications.
It's probably highly workload dependent. If most of cached items are reused
frequently and there is not enough memory, Solr may need to keep recreation and
uses more CPU cycles. However, if cached items are not reused too frequently,
the change can help.
I agree with you that there is performance implications. The WeakHashMap idea
is basically a tradeoff, trading more CPU cycles for lower memory footprint, as
mentioned in JIRA description. That's why I propose it to be optional. It's
intended to be a tuning knob for user to decide depending on their use cases.
> Remove strong reference to Field Cache key (optional) so that GC can release
> some Field Cache entries when Solr is under memory pressure
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-10015
> URL: https://issues.apache.org/jira/browse/SOLR-10015
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Michael Sun
> Attachments: SOLR-10015-prototype.patch
>
>
> In current Field Cache (FC) implementation, a WeakHashMap is used, supposedly
> designed to allow GC to release some Field Cache entries when Solr is under
> memory pressure.
> However, in practice, FC entry releasing seldom happens. Even worse, sometime
> Solr goes OOM and heap dump shows large amount of memory is actually used by
> FC. It's a sign that GC is not able to release FC entries even WeakHashMap is
> used.
> The reason is that FC is using SegmentCoreReaders as the key to the
> WeakHashMap. However, SegmentCoreReaders is usually strong referenced by
> SegmentReader. A strong reference would prevent GC to release the key and
> therefore the value. Therefore GC can't release entries in FC's WeakHashMap.
> The JIRA is to propose a solution to remove the strong reference mentioned
> above so that GC can release FC entries to avoid long GC pause or OOM. It
> needs to be optional because this change is a tradeoff, trading more CPU
> cycles for low memory footage. User can make final decision depending on
> their use cases.
> The prototype attached use a combination of directory name and segment name
> as key to the WeakHashMap, replacing the SegmentCoreReaders. Without change,
> Solr doesn't release any FC entries after a GC is manually triggered. With
> the change, FC entries are usually released after GC.
> However, I am not sure if it's the best way to solve this problem. Any
> suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
