Yukang-Lian opened a new pull request, #37379: URL: https://github.com/apache/doris/pull/37379
Pick [Fix](group commit) Fix group commit block queue mem estimate faule #35314 ## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> **Problem:** When `group commit=async_mode` and NULL data is imported into a `variant` type column, it causes incorrect memory statistics for group commit backpressure, leading to a stuck issue. **Cause:** In group commit mode, blocks are first added to a queue in batches using `add block`, and then blocks are retrieved from the queue using `get block`. To track memory usage during backpressure, we add the block size to the memory statistics during `add block` and subtract the block size from the memory statistics during `get block`. However, for `variant` types, during the `add block` write to WAL, serialization occurs, which can merge types (e.g., merging `int` and `bigint` into `bigint`), thereby changing the block size. This results in a discrepancy between the block size during `get block` and `add block`, causing memory statistics to overflow. **Solution:** Record the block size at the time of `add block` and use this recorded size during `get block` instead of the actual block size. This ensures consistency in the memory addition and subtraction. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... ## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
