[
https://issues.apache.org/jira/browse/SOLR-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839958#comment-15839958
]
Yonik Seeley commented on SOLR-10015:
-------------------------------------
Removing the strong references means that cache entries that are hugely
expensive to build can be dropped without warning (and not even in response to
memory pressure... just dropped because of any old GC event).
Small changes from test to production could lead to wildly different behavior.
If we want more control over field cache memory, it makes more sense to create
a bounded cache (like other Solr caches) that can be controlled by size (number
of elements or RAM usage).
bq. In current Field Cache (FC) implementation, a WeakHashMap is used,
supposedly designed to allow GC to release some Field Cache entries when Solr
is under memory pressure.
That was never it's purpose (and it would not work). A WeakHashMap was used to
prevent actual memory leaks, nothing more. It's just because of how the
FieldCache was originally designed in Lucene... it was decoupled, so a reader
had no knowledge that there were associated cache entries. The system
(IndexReader+FieldCache) as a whole was designed as an unbounded cache. Once
an entry was instantiated for a reader, it was never removed until the reader
itself went away.
> Remove strong reference to Field Cache key (optional) so that GC can release
> some Field Cache entries when Solr is under memory pressure
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-10015
> URL: https://issues.apache.org/jira/browse/SOLR-10015
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Michael Sun
> Attachments: SOLR-10015-prototype.patch
>
>
> In current Field Cache (FC) implementation, a WeakHashMap is used, supposedly
> designed to allow GC to release some Field Cache entries when Solr is under
> memory pressure.
> However, in practice, FC entry releasing seldom happens. Even worse, sometime
> Solr goes OOM and heap dump shows large amount of memory is actually used by
> FC. It's a sign that GC is not able to release FC entries even WeakHashMap is
> used.
> The reason is that FC is using SegmentCoreReaders as the key to the
> WeakHashMap. However, SegmentCoreReaders is usually strong referenced by
> SegmentReader. A strong reference would prevent GC to release the key and
> therefore the value. Therefore GC can't release entries in FC's WeakHashMap.
> The JIRA is to propose a solution to remove the strong reference mentioned
> above so that GC can release FC entries to avoid long GC pause or OOM. It
> needs to be optional because this change is a tradeoff, trading more CPU
> cycles for low memory footage. User can make final decision depending on
> their use cases.
> The prototype attached use a combination of directory name and segment name
> as key to the WeakHashMap, replacing the SegmentCoreReaders. Without change,
> Solr doesn't release any FC entries after a GC is manually triggered. With
> the change, FC entries are usually released after GC.
> However, I am not sure if it's the best way to solve this problem. Any
> suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
