LiuBiao created FLINK-4164:
------------------------------
Summary: Use only one rocksdb column family to store all states in
an operator
Key: FLINK-4164
URL: https://issues.apache.org/jira/browse/FLINK-4164
Project: Flink
Issue Type: Improvement
Components: State Backends, Checkpointing
Reporter: LiuBiao
Priority: Minor
I see now in master branch, there is only one rocksdb instance in an operator,
each state will be assigned into separated column families. It's a good
improvement, but I think there are still some problems. Column families in
rocksdb do not share memtable and sst files(correct me if I am wrong, there is
not much docs about column families in rocksdb). If user have thousands of
states, the overhead will be costly. Memory may will be exhausted, also there
will be too much small sst files.
If we use prefix in key instead of column family, the memory that rocksdb
costed will be under control. The shortcoming is that we can not optimize
states separately, and performance may be a little worse than before. But I
think it is worth to do, it is meaning for some cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
