bxfjb opened a new issue, #8017: URL: https://github.com/apache/rocketmq/issues/8017
### Before Creating the Bug Report - [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment CentOS 7 ### RocketMQ version branch: develop ### JDK Version OpenJDK 8 ### Describe the Bug We deployed a rocketmq cluster with TieredStore and offload cold data to hdd with PosixFileSegment. After about 16 hours of producing messages, the shutdownHook was called and there is no other error log. > 2024-04-11 09:57:22 INFO ShutdownHook - Shutdown hook was invoked, 1 By checking monitor, we found out that the direct memory usage kept on increasing in a stable rate,  which reminds us of checking if there is memory leak. Actually the increase rate is a direct clue -- about 570 MB each time, exactly the size of a compacted IndexFile, which leads us to here at `IndexStoreService.java`: ``` public boolean doCompactThenUploadFile(IndexFile indexFile) { ... boolean result = flatAppendFile.commitAsync().join(); ... } ``` Add a watch of direct memory when debugging the UT `doCompactionTest` at `IndexStoreServiceTest.java` can prove that `doCompactThenUploadFile` may cause direct memory leak. By diving in the code, the root cause actually is inappropriate use of FileChannel at `PosixFileSegment.java`: ``` @Override @SuppressWarnings("ResultOfMethodCallIgnored") public CompletableFuture<Boolean> commit0( FileSegmentInputStream inputStream, long position, int length, boolean append) { return CompletableFuture.supplyAsync(() -> { try { byte[] byteArray = ByteStreams.toByteArray(inputStream); writeFileChannel.position(position); ByteBuffer buffer = ByteBuffer.wrap(byteArray); while (buffer.hasRemaining()) { writeFileChannel.write(buffer); } writeFileChannel.force(true); } catch (Exception e) { return false; } return true; }, MessageStoreExecutor.getInstance().bufferCommitExecutor); } ``` The line `writeFileChannel.write(buffer);` would allocate a DirectByteBuffer whose size is same with `buffer`, which means in uploading of IndexFile, the DirectByteBuffer's size will be about 570MB. This DirectByteBuffer is an element of an array of ByteBuffer of a ThreadLocal variable in : ### Steps to Reproduce 2 choices: 1. Deploy a rocketmq cluster with TieredStore and offload cold data to hdd with PosixFileSegment, keep on producing new messages and the direct memory usage should increase unlimitedly. 2. Or run IndexStoreServiceTest.doCompactionTest and keep an eye on the direct buffer usage. ### What Did You Expect to See? The direct memory usage should not increase unlimitedly. ### What Did You See Instead? The direct memory usage keeps on increasing until OOM or all threads in commitExecutor run commit0 at least once. ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
