[
https://issues.apache.org/jira/browse/FLINK-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009811#comment-17009811
]
Yun Tang commented on FLINK-15368:
----------------------------------
Thanks for [~sewen] comments. I think even we increase the managed memory to
match the previously dedicated memory, the mechanism of write buffer manager
would still affect the performance as write buffer is flushed judged by the
reserved memory not the actually used memory.
Below is my two cents of the last two questions:
The reason why we choose 0.5 as default write buffer manager ratio is for easy
to understand: half write vs half read. I think we could also increase that
default ratio to a larger value like 0.8 to simulate the default behavior for
RocksDB (64MB write buffer * 2 vs 8MB cache).
I believe fewer column families would help reduce the memory usage. The main
challenge we face is the performance regression once we use only one column
family. There exists a trade off between controable memory usage vs performance.
> Add end-to-end test for controlling RocksDB memory usage
> --------------------------------------------------------
>
> Key: FLINK-15368
> URL: https://issues.apache.org/jira/browse/FLINK-15368
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Affects Versions: 1.10.0
> Reporter: Yu Li
> Assignee: Yun Tang
> Priority: Critical
> Fix For: 1.10.0
>
>
> We need to add an end-to-end test to make sure the RocksDB memory usage
> control works well, especially under the slot sharing case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
