[
https://issues.apache.org/jira/browse/SOLR-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114059#comment-18114059
]
ASF subversion and git services commented on SOLR-18415:
--------------------------------------------------------
Commit 33949044b2c92bb137d02546ac0cbd030e8733a2 in solr's branch
refs/heads/branch_10x from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=33949044b2c ]
SOLR-18415: CrossDC Common: improve serialization for large docs. (#4887)
(cherry picked from commit ba8f866d05371bc54898ad93ec06cf2f998d05ac)
> CrossDC Common: ExposedByteArrayOutputStream poor optimization for large docs
> -----------------------------------------------------------------------------
>
> Key: SOLR-18415
> URL: https://issues.apache.org/jira/browse/SOLR-18415
> Project: Solr
> Issue Type: Bug
> Components: module - crossDC
> Affects Versions: 10.0, 9.10.1
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Major
> Labels: pull-request-available
> Fix For: main(11.0), 10.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{MirroredSolrRequestSerializer}} uses a subclass of
> {{ByteArrayOutputStream}} in an attempt to optimize data transfer by avoiding
> array copy - instead it passes the whole buffer directly as the output bytes
> from {{{}serialize(){}}}.
> This sounds like a plausible optimization. However, the initial size of the
> buffer (32 bytes) is doubled on each overflow as the codec writes to the
> stream, which means that on average the actual serialized data occupies
> around half of the buffer, the rest are zeroes. Deserialization will handle
> these trailing zeros just fine, it's the cost of transmitting roughly twice
> as much data over the wire that is questionable - this buffer is then passed
> as-is to the output stream and eventually ends up as network traffic.
> This "optimization" avoids one array copy cost (which reduces GC) but
> generates multiple costs on the way - not only the empty data has to be sent
> over the network but also it needs to be stored in source Kafka, in
> MirrorMaker and in target Kafka, to be finally retrieved by the Consumer
> (which has to allocate a buffer sufficiently large to fit also the zeroes),
> only to discard the zeroes during deserialization.
> I propose to get rid of this optimization and use the plain
> {{ByteArrayOutputStream}} instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
