[
https://issues.apache.org/jira/browse/KUDU-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289197#comment-16289197
]
ZhangZhen commented on KUDU-2238:
---------------------------------
I found the DMS which consumes a lot memory also has a large log retention
number, so to work around this problem, I changed the config
log_target_replay_size_mb from default 1024 to 250 to remind MaintenanceManager
flush it. The config works and the DMS is flushed finally after I restart the
tablet server.
> Big DMS not flush under memory pressure
> ---------------------------------------
>
> Key: KUDU-2238
> URL: https://issues.apache.org/jira/browse/KUDU-2238
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 1.3.0
> Environment: CentOS6.5 Linux 2.6.32-431
> Kudu1.3.0
> GitCommit 00813f96b9cb
> Reporter: ZhangZhen
> Attachments: memory_anchored.png, memory_consumed.png
>
>
> I have a table with many updates, its DMS consumes a lot memory and cause
> “Soft Memory Limit Exceed”. I check the /mem-trackers on the tablet server,
> one of its DMS consumes about 3G memory, but check the /maintenance-manager,
> its FlushDeltaMemStoresOp can only free 763B anchored memory and
> perf_improvement is 0. Is this normal? I know Kudu is not optimized for
> updates, but still confused why the DMS won’t be flushed under memory
> pressure.
> Infos from /mem-trackers:
> !memory_consumed.png!
> tablet-5941a8bb934e4686abd1bfff9e35c860 server none 3.00G 3.00G
> txn_tracker tablet-5941a8bb934e4686abd1bfff9e35c860 64.00M 0B
> 1.67M
> MemRowSet-339 tablet-5941a8bb934e4686abd1bfff9e35c860 none 265B
> 265B
> DeltaMemStores tablet-5941a8bb934e4686abd1bfff9e35c860 none 3.00G
> 3.00G
> Infos from /maintenance-manager
> !memory_anchored.png!
> FlushDeltaMemStoresOp(5941a8bb934e4686abd1bfff9e35c860) true 763B
> 511.15M 0
> The tablet 5941a8bb934e4686abd1bfff9e35c860 has 16 RowSets in total
> Some configs of MM:
> --enable_maintenance_manager=true
> --log_target_replay_size_mb=1024
> --maintenance_manager_history_size=8
> --maintenance_manager_num_threads=6
> --maintenance_manager_polling_interval_ms=50
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
