xinyiZzz opened a new pull request, #11771:
URL: https://github.com/apache/doris/pull/11771
# Proposed changes
Issue Number: close #11770
## Problem summary
1. Reason:
The memory value automatically tracked by the tcmalloc hook in the
DeltaWriter is smaller than the value recorded manually in the memtable,
because the first 4096-byte Chunk requested by each MemPool when the memtable
is initialized is not recorded in the DeltaWriter by the hook.
The values โโof the two are not equal, causing the `mem_consumption() ==
_mem_table->memory_usage ` branch judgment to fail.
โ
2. Solution:
In order to completely avoid the modification of mem tracker and affect the
judgment of `mem_consumption() == _mem_table->memory_usage ` branch, change it
to ` if (_flush_token->get_stats().flush_running_count == 0) `,
flush_running_count is counted in FlushToken and divided Open two kinds of
flush memory statistics logic.
Memory value automatically tracked by tcmalloc hook in DeltaWriter for load
channel
Flush memtable is triggered when the mem limit is exceeded. The manually
recorded value of the memtable is used to flush the memtable when it is larger
than write_buffer_size.
โ
3. Can use independent variables to control flush:
At present, when the flush is larger than write_buffer_size, it is already
manually controlled independently;
When the flush is larger than the mem limit, the tcmallloc hook is still
used for automatic statistics, and the selection of the flush DeltaWriter is
realized based on the hierarchical relationship of the mem tracker.
Currently, the memory value automatically tracked by the tcmalloc hook is
20% less than the manually recorded value in the memtable, because some of the
freed memory is not allocated by the DeltaWriter. `
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
