[
https://issues.apache.org/jira/browse/FLINK-31225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaogang zhou updated FLINK-31225:
----------------------------------
Description:
the default value for
state.backend.rocksdb.files.open
is -1
[https://github.com/facebook/rocksdb/issues/4112] this issue told us the
rocksdb will not close fd , some this can lead to oom issue.
also I can reproduce the situation in my enviroment. left part(2/21- 2/24) is
leave max open file to -1, right part(2/24 till now) is set the max open file
to 300. the memory grow is very differnt.
!image-2023-02-26-12-08-49-717.png|width=616,height=285!
I have also attached a jeprof for 2/21-2/24 instance, the tm memory size is
about 8GB, heap memory is about 2.6GB, the native part in leak_test is about
1GB. I think the remaining part (8-2.6-1)is occupied by fd .
I suggest set this to a default value like 500.
was:
the default value for
state.backend.rocksdb.files.open
is -1
[https://github.com/facebook/rocksdb/issues/4112] this issue told us the
rocksdb will not close fd , some this can lead to oom issue.
also I can reproduce the situation in my enviroment. left part(2/21- 2/24) is
leave max open file to -1, right part(2/24 till now) is set the max open file
to 300. the memory grow is very differnt.
!image-2023-02-26-12-08-49-717.png|width=616,height=285!
I suggest set this to a default value like 500.
> rocksdb max open file can lead to oom
> --------------------------------------
>
> Key: FLINK-31225
> URL: https://issues.apache.org/jira/browse/FLINK-31225
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.16.1
> Reporter: xiaogang zhou
> Priority: Major
> Attachments: image-2023-02-26-12-08-49-717.png, leak_test(2).png
>
>
> the default value for
> state.backend.rocksdb.files.open
> is -1
>
> [https://github.com/facebook/rocksdb/issues/4112] this issue told us the
> rocksdb will not close fd , some this can lead to oom issue.
>
> also I can reproduce the situation in my enviroment. left part(2/21- 2/24) is
> leave max open file to -1, right part(2/24 till now) is set the max open file
> to 300. the memory grow is very differnt.
> !image-2023-02-26-12-08-49-717.png|width=616,height=285!
>
> I have also attached a jeprof for 2/21-2/24 instance, the tm memory size is
> about 8GB, heap memory is about 2.6GB, the native part in leak_test is about
> 1GB. I think the remaining part (8-2.6-1)is occupied by fd .
>
> I suggest set this to a default value like 500.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
