[
https://issues.apache.org/jira/browse/SOLR-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089648#comment-18089648
]
ASF subversion and git services commented on SOLR-18237:
--------------------------------------------------------
Commit c4b69c2d92bdbbaffc9a34c8fad335a50179a1dd in solr's branch
refs/heads/branch_10x from Pierre Salagnac
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=c4b69c2d92b ]
SOLR-18237: Do not copy data before noggit parsing (#4415)
This removes the char[] array buffer when parsing collection states and other
Json data, since allocation of this large buffer is a cause of OOM errors.
Instead, the UTF8 decoding is done on the fly using JVM tolling.
(cherry picked from commit 2c8cb3235e16f3b0c45df50c3ceb218d093095d4)
> Collection state deserialization duplicates content for no good reason
> ----------------------------------------------------------------------
>
> Key: SOLR-18237
> URL: https://issues.apache.org/jira/browse/SOLR-18237
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10, 10.0
> Reporter: Pierre Salagnac
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When deserializing collection state from Zookeeper, we create large char
> arrays to convert UTF8 data into Java strings.
> In Utils.java
> {code:java}
> CharArr chars = new CharArr();
> ByteUtils.UTF8toUTF16(utf8, offset, length, chars);
> {code}
> This consumes a lot of memory several threads concurrently deserialize a big
> collection. We had OOM error because of this. This is a corner case since the
> OOM were raised with a non full heap (~50%). My understanding is this is
> because big arrays are allocated directly in the _Old_ space, and we end with
> fragmented heap where one more big array does not fit, even if the heap is
> not full.
>
> This may easily replaced by a reader that does the UTF8 decoding on the fly,
> with a much much smaller buffer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
