[
https://issues.apache.org/jira/browse/SOLR-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Salagnac resolved SOLR-18237.
------------------------------------
Fix Version/s: 10.1
9.11
Resolution: Fixed
> Collection state deserialization duplicates content for no good reason
> ----------------------------------------------------------------------
>
> Key: SOLR-18237
> URL: https://issues.apache.org/jira/browse/SOLR-18237
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10, 10.0
> Reporter: Pierre Salagnac
> Priority: Major
> Labels: pull-request-available
> Fix For: 10.1, 9.11
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When deserializing collection state from Zookeeper, we create large char
> arrays to convert UTF8 data into Java strings.
> In Utils.java
> {code:java}
> CharArr chars = new CharArr();
> ByteUtils.UTF8toUTF16(utf8, offset, length, chars);
> {code}
> This consumes a lot of memory several threads concurrently deserialize a big
> collection. We had OOM error because of this. This is a corner case since the
> OOM were raised with a non full heap (~50%). My understanding is this is
> because big arrays are allocated directly in the _Old_ space, and we end with
> fragmented heap where one more big array does not fit, even if the heap is
> not full.
>
> This may easily replaced by a reader that does the UTF8 decoding on the fly,
> with a much much smaller buffer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
