[
https://issues.apache.org/jira/browse/KUDU-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18061431#comment-18061431
]
ASF subversion and git services commented on KUDU-3734:
-------------------------------------------------------
Commit 63d9c39267bc7f35dc3a2e7535148b28088b229a in kudu's branch
refs/heads/master from Ashwani
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=63d9c3926 ]
KUDU-3734 include undo size while picking rowsets
As part of compaction improvement effort, this change is focussing on
taking into consideration the size of undo deltas while picking rowsets
during rowset compaction. I could not find any historical reason to why
it was not done before. Maybe there was some analysis done that ended
in a conclusion that considering undo deltas may not be right approach
when estimating upper bound fractional solution in the knapsack. The
size of undo deltas are taken into consideration while deciding whether
a new rowset (a potential compaction 'candidate') fits into the budget
and is at least denser than least dense candidate in knapsack.
The test data used here has significant undo deltas that fits perfectly
into the OOM scenario. Total size of uncompacted data is 29GB.
With this patch, rowset compaction never hit OOM and the resident
memory kept well within limits (~1GB).
Without this patch, compaction hit OOM on a node with limited memory
where rowset compaction peak memory touched ~30GB.
Testing with different scenarios:
1. Compaction that takes into consideration:
- Base data
- Redo deltas
- Undo delats
Budget for compaction (tablet_compaction_budget_mb) is 1024 MB
Result: Rowset 211 was skipped due to 1024 MB size constraint.
Budgeted compaction selection:
[ ] RowSet(211)( 1945M) [0.0000, 1.0000]
[ ] RowSet(217)( 1M) [0.0027, 0.1953]
[x] RowSet(216)( 1M) [0.5341, 0.5341]
[x] RowSet(218)( 1M) [0.5341, 0.5341]
[x] RowSet(219)( 1M) [0.5341, 0.5341]
2. Compaction that takes into consideration:
- Base data
- Redo deltas
- Undo delats
Budget for compaction (tablet_compaction_budget_mb) is 2048 MB
Result: Rowset 211 was NOT skipped with 2048 MB size limit.
Budgeted compaction selection:
[x] RowSet(211)( 1945M) [0.0000, 1.0000]
[x] RowSet(217)( 1M) [0.0027, 0.1953]
[x] RowSet(216)( 1M) [0.5341, 0.5341]
[x] RowSet(218)( 1M) [0.5341, 0.5341]
[x] RowSet(219)( 1M) [0.5341, 0.5341]
3. Compaction that takes into consideration:
- Base data
- Redo deltas
Compaction budget (tablet_compaction_budget_mb) is default 128 MB
Result: Rowset 211 with size 1M (ignoring UNDO deltas) included.
Budgeted compaction selection:
[x] RowSet(211)( 1M) [0.0000, 1.0000]
[x] RowSet(217)( 1M) [0.0026, 0.2015]
[x] RowSet(216)( 1M) [0.5358, 0.5385]
[x] RowSet(218)( 1M) [0.5385, 0.5404]
[x] RowSet(219)( 1M) [0.5404, 0.5404]
Note: This is different from rowset compaction batching effort.
Change-Id: I351c0ba96a02e6ded5153adf9d981083a8c40592
Reviewed-on: http://gerrit.cloudera.org:8080/23348
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
> Rowset merge compaction does not consider undo delta size while picking
> rowsets
> -------------------------------------------------------------------------------
>
> Key: KUDU-3734
> URL: https://issues.apache.org/jira/browse/KUDU-3734
> Project: Kudu
> Issue Type: Bug
> Reporter: Ashwani Raina
> Assignee: Ashwani Raina
> Priority: Major
>
> Rowset merge compaction has preliminary stage where it goes through all the
> eligible rowsets in a tablet that fit the criteria of optimal compaction
> w.r.t., budget, rowset width, range overlaps, etc.
> It uses fractional knapsack algorithm to come up with the list of rowsets
> that can deliver maximum bang for the buck. While a rowset merge compaction
> touches almost all the parts of a data pertaining to the rows within a
> tablet, e.g., base data, redo and undo deltas, it doesn't take into
> consideration the size of undo deltas while calculating density of a rowset
> item in knapsack algorithm.
> Due to this, even if a rowset has undo deltas amount to huge size, it is
> possible that it will get picked up for rowset merge compaction. This can
> lead to OOM scenarios if say size of undo deltas is GBs and when compaction
> finally starts reading the rows and corresponding uncompressed data from
> deltas into memory, the process might cross the memory hard limit, thereby
> forcing OS to take action and kill the kudu service. One such example is
> KUDU-3406.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
