[
https://issues.apache.org/jira/browse/IGNITE-29206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29206:
-----------------------------------
Summary: DirectMessageWriter retains oversized temporary serialization
buffer and reuses it after a failure (was: DirectMessageWriter retains
oversized temporary serialization buffer per connection)
> DirectMessageWriter retains oversized temporary serialization buffer and
> reuses it after a failure
> --------------------------------------------------------------------------------------------------
>
> Key: IGNITE-29206
> URL: https://issues.apache.org/jira/browse/IGNITE-29206
> Project: Ignite
> Issue Type: Improvement
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
>
> *Problem*
> DirectMessageWriter serializes @Compress fields and pending raw fields into a
> reusable heap scratch buffer (tmpBuf). The data is then compressed or copied
> out of it. Three things together cause the problem:
> - The buffer starts at 10 KB and doubles until the whole field fits.
> - It is never shrunk.
> - The nested tmpWriter streams keep a reference to it too.
> A writer lives as long as its NIO session, because it is stored in the
> session meta. So every connection ends up holding a buffer as large as the
> biggest field it has ever serialized, up to twice that field's raw size.
> The exchange coordinator is hit hardest. It sends
> GridDhtPartitionsFullMessage to every node. On large clusters, compressed
> fields such as locParts can reach tens of megabytes of raw data. After a PME,
> each of the N communication sessions keeps such a buffer. That adds up to
> potentially hundreds of megabytes on large clusters of heap that is never
> released. Other nodes are affected the same way by
> GridDhtPartitionsSingleMessage.
> The read side (DirectMessageReader) does not have this issue. The
> decompressed payload is allocated per message, and releasePayload() resets
> the stream references once it has been read.
> *Fix*
> - Once the temporary buffer has been used, release it if its capacity is
> above 256 KB. "Used" means deflated into the CompressedMessage chunks or
> copied into a RawField.
> - On release, also reset the nested tmpWriter and all its state-item streams
> to a shared empty buffer, so nothing keeps the large array reachable. This
> mirrors DirectMessageReader#releasePayload().
> - Buffers up to 256 KB are still reused, so small and medium fields keep
> serializing with no per-message allocations.
> - An oversized field now grows a fresh buffer each time. That copying is
> negligible compared to serializing and compressing such a field.
> - The wire format is unchanged.
> *Tests*
> - New test CompressedMessageTest#testLargeTemporaryBufferReleased. It writes
> a small message, then a large GridDhtPartitionsFullMessage twice, through the
> same writer. After each write it checks that neither the writer nor its
> tmpWriter streams hold a buffer above the limit, and that the message reads
> back correctly. The test fails without the fix.
> - Existing serialization tests pass: CompressedMessageTest,
> DirectMarshallingMessagesTest, RollingUpgradeMessageSerializationTest,
> *MessageSerializationTest.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
