Andrzej Bialecki created SOLR-18415:
---------------------------------------
Summary: CrossDC Common: ExposedByteArrayOutputStream optimizes
wrong aspect
Key: SOLR-18415
URL: https://issues.apache.org/jira/browse/SOLR-18415
Project: Solr
Issue Type: Bug
Components: module - crossDC
Affects Versions: 9.10.1, 10.0
Reporter: Andrzej Bialecki
Assignee: Andrzej Bialecki
{{MirroringSolrRequestSerializer}} uses a subclass of {{ByteArrayOutputStream}}
in an attempt to optimize data transfer by avoiding array copy - instead it
passes the whole buffer directly as the output bytes from {{{}serialize(){}}}.
This sounds like a plausible optimization. However, the initial size of the
buffer (32 bytes) is doubled on each overflow as the codec writes to the
stream, which means that on average the actual serialized data occupies around
half of the buffer, the rest are zeroes. Deserialization will handle these
trailing zeros just fine, it's the cost of transmitting roughly twice as much
data over the wire that is questionable - this buffer is then passed as-is to
the output stream and eventually ends up as network traffic.
This "optimization" avoids one array copy cost (which reduces GC) but generates
multiple costs on the way - not only the empty data has to be sent over the
network but also it needs to be stored in source Kafka, in MirrorMaker and in
target Kafka, to be finally retrieved by the Consumer (which has to allocate a
buffer sufficiently large to fit also the zeroes), only to discard the zeroes
during deserialization.
I propose to get rid of this optimization and use the plain
{{ByteArrayOutputStream}} instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
