Dmitry Werner created IGNITE-29206:
--------------------------------------
Summary: DirectMessageWriter retains oversized temporary
serialization buffer per connection
Key: IGNITE-29206
URL: https://issues.apache.org/jira/browse/IGNITE-29206
Project: Ignite
Issue Type: Improvement
Reporter: Dmitry Werner
Assignee: Dmitry Werner
Problem
DirectMessageWriter serializes @Compress fields and pending raw fields into a
reusable heap scratch buffer (tmpBuf). The data is then compressed or copied
out of it. Three things together cause the problem:
- The buffer starts at 10 KB and doubles until the whole field fits.
- It is never shrunk.
- The nested tmpWriter streams keep a reference to it too.
A writer lives as long as its NIO session, because it is stored in the session
meta. So every connection ends up holding a buffer as large as the biggest
field it has ever serialized, up to twice that field's raw size.
The exchange coordinator is hit hardest. It sends GridDhtPartitionsFullMessage
to every node. On large clusters, compressed fields such as locParts can reach
tens of megabytes of raw data. After a PME, each of the N communication
sessions keeps such a buffer. That adds up to potentially hundreds of megabytes
on large clusters of heap that is never released. Other nodes are affected the
same way by GridDhtPartitionsSingleMessage.
The read side (DirectMessageReader) does not have this issue. The decompressed
payload is allocated per message, and releasePayload() resets the stream
references once it has been read.
Fix
- Once the temporary buffer has been used, release it if its capacity is above
256 KB. "Used" means deflated into the CompressedMessage chunks or copied into
a RawField.
- On release, also reset the nested tmpWriter and all its state-item streams to
a shared empty buffer, so nothing keeps the large array reachable. This mirrors
DirectMessageReader#releasePayload().
- Buffers up to 256 KB are still reused, so small and medium fields keep
serializing with no per-message allocations.
- An oversized field now grows a fresh buffer each time. That copying is
negligible compared to serializing and compressing such a field.
- The wire format is unchanged.
Tests
- New test CompressedMessageTest#testLargeTemporaryBufferReleased. It writes a
small message, then a large GridDhtPartitionsFullMessage twice, through the
same writer. After each write it checks that neither the writer nor its
tmpWriter streams hold a buffer above the limit, and that the message reads
back correctly. The test fails without the fix.
- Existing serialization tests pass: CompressedMessageTest,
DirectMarshallingMessagesTest, RollingUpgradeMessageSerializationTest,
*MessageSerializationTest.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
