[
https://issues.apache.org/jira/browse/IGNITE-29206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29206:
-----------------------------------
Description:
*Problem*
DirectMessageWriter serializes @Compress fields and pending raw fields into a
reusable heap scratch buffer (tmpBuf). The data is then compressed or copied
out of it. Three things together cause the problem:
- The buffer starts at 10 KB and doubles until the whole field fits.
- It is never shrunk.
- The nested tmpWriter streams keep a reference to it too.
A writer lives as long as its NIO session, because it is stored in the session
meta. So every connection ends up holding a buffer as large as the biggest
field it has ever serialized, up to twice that field's raw size.
The exchange coordinator is hit hardest. It sends GridDhtPartitionsFullMessage
to every node. On large clusters, compressed fields such as locParts can reach
tens of megabytes of raw data. After a PME, each of the N communication
sessions keeps such a buffer. That adds up to potentially hundreds of megabytes
on large clusters of heap that is never released. Other nodes are affected the
same way by GridDhtPartitionsSingleMessage.
The read side (DirectMessageReader) does not have this issue. The decompressed
payload is allocated per message, and releasePayload() resets the stream
references once it has been read.
*Fix*
- Once the temporary buffer has been used, release it if its capacity is above
256 KB. "Used" means deflated into the CompressedMessage chunks or copied into
a RawField.
- On release, also reset the nested tmpWriter and all its state-item streams
to a shared empty buffer, so nothing keeps the large array reachable. This
mirrors DirectMessageReader#releasePayload().
- Buffers up to 256 KB are still reused, so small and medium fields keep
serializing with no per-message allocations.
- An oversized field now grows a fresh buffer each time. That copying is
negligible compared to serializing and compressing such a field.
- The wire format is unchanged.
*Tests*
- New test CompressedMessageTest#testLargeTemporaryBufferReleased. It writes a
small message, then a large GridDhtPartitionsFullMessage twice, through the
same writer. After each write it checks that neither the writer nor its
tmpWriter streams hold a buffer above the limit, and that the message reads
back correctly. The test fails without the fix.
- Existing serialization tests pass: CompressedMessageTest,
DirectMarshallingMessagesTest, RollingUpgradeMessageSerializationTest,
*MessageSerializationTest.
was:
Problem
DirectMessageWriter serializes @Compress fields and pending raw fields into a
reusable heap scratch buffer (tmpBuf). The data is then compressed or copied
out of it. Three things together cause the problem:
- The buffer starts at 10 KB and doubles until the whole field fits.
- It is never shrunk.
- The nested tmpWriter streams keep a reference to it too.
A writer lives as long as its NIO session, because it is stored in the session
meta. So every connection ends up holding a buffer as large as the biggest
field it has ever serialized, up to twice that field's raw size.
The exchange coordinator is hit hardest. It sends GridDhtPartitionsFullMessage
to every node. On large clusters, compressed fields such as locParts can reach
tens of megabytes of raw data. After a PME, each of the N communication
sessions keeps such a buffer. That adds up to potentially hundreds of megabytes
on large clusters of heap that is never released. Other nodes are affected the
same way by GridDhtPartitionsSingleMessage.
The read side (DirectMessageReader) does not have this issue. The decompressed
payload is allocated per message, and releasePayload() resets the stream
references once it has been read.
Fix
- Once the temporary buffer has been used, release it if its capacity is above
256 KB. "Used" means deflated into the CompressedMessage chunks or copied into
a RawField.
- On release, also reset the nested tmpWriter and all its state-item streams to
a shared empty buffer, so nothing keeps the large array reachable. This mirrors
DirectMessageReader#releasePayload().
- Buffers up to 256 KB are still reused, so small and medium fields keep
serializing with no per-message allocations.
- An oversized field now grows a fresh buffer each time. That copying is
negligible compared to serializing and compressing such a field.
- The wire format is unchanged.
Tests
- New test CompressedMessageTest#testLargeTemporaryBufferReleased. It writes a
small message, then a large GridDhtPartitionsFullMessage twice, through the
same writer. After each write it checks that neither the writer nor its
tmpWriter streams hold a buffer above the limit, and that the message reads
back correctly. The test fails without the fix.
- Existing serialization tests pass: CompressedMessageTest,
DirectMarshallingMessagesTest, RollingUpgradeMessageSerializationTest,
*MessageSerializationTest.
> DirectMessageWriter retains oversized temporary serialization buffer per
> connection
> -----------------------------------------------------------------------------------
>
> Key: IGNITE-29206
> URL: https://issues.apache.org/jira/browse/IGNITE-29206
> Project: Ignite
> Issue Type: Improvement
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
>
> *Problem*
> DirectMessageWriter serializes @Compress fields and pending raw fields into a
> reusable heap scratch buffer (tmpBuf). The data is then compressed or copied
> out of it. Three things together cause the problem:
> - The buffer starts at 10 KB and doubles until the whole field fits.
> - It is never shrunk.
> - The nested tmpWriter streams keep a reference to it too.
> A writer lives as long as its NIO session, because it is stored in the
> session meta. So every connection ends up holding a buffer as large as the
> biggest field it has ever serialized, up to twice that field's raw size.
> The exchange coordinator is hit hardest. It sends
> GridDhtPartitionsFullMessage to every node. On large clusters, compressed
> fields such as locParts can reach tens of megabytes of raw data. After a PME,
> each of the N communication sessions keeps such a buffer. That adds up to
> potentially hundreds of megabytes on large clusters of heap that is never
> released. Other nodes are affected the same way by
> GridDhtPartitionsSingleMessage.
> The read side (DirectMessageReader) does not have this issue. The
> decompressed payload is allocated per message, and releasePayload() resets
> the stream references once it has been read.
> *Fix*
> - Once the temporary buffer has been used, release it if its capacity is
> above 256 KB. "Used" means deflated into the CompressedMessage chunks or
> copied into a RawField.
> - On release, also reset the nested tmpWriter and all its state-item streams
> to a shared empty buffer, so nothing keeps the large array reachable. This
> mirrors DirectMessageReader#releasePayload().
> - Buffers up to 256 KB are still reused, so small and medium fields keep
> serializing with no per-message allocations.
> - An oversized field now grows a fresh buffer each time. That copying is
> negligible compared to serializing and compressing such a field.
> - The wire format is unchanged.
> *Tests*
> - New test CompressedMessageTest#testLargeTemporaryBufferReleased. It writes
> a small message, then a large GridDhtPartitionsFullMessage twice, through the
> same writer. After each write it checks that neither the writer nor its
> tmpWriter streams hold a buffer above the limit, and that the message reads
> back correctly. The test fails without the fix.
> - Existing serialization tests pass: CompressedMessageTest,
> DirectMarshallingMessagesTest, RollingUpgradeMessageSerializationTest,
> *MessageSerializationTest.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
