Re: [DISCUSS] FLIP-397: Add config options for administrator JVM options

2023-12-29 Thread xiangyu feng
Hi Zhanghao, Thanks for driving this. +1 for the overall idea. One minor question, do we need separate administrator JVM options for both JobManager and TaskManager? Or just one administrator JVM option for all? I'm afraid of 6 jvm options(env.java.opts.all\env.java.default-opts.all\env.java.opt

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

2023-12-29 Thread xiangyu feng
Thanks Yangze for restart this discussion. +1 for the overall idea. By splitting the HighAvailabilityServices into LeaderServices and PersistenceServices, we may support configuring different storage behind them in the future. We did run into real problems in production where too much job metadat

回复: Anyone encounters the non-heap memory increasing issue when using RocksDB as state backend and has better solutions?

2023-12-29 Thread Sue Alen
Hi, Zakelly, It’s Great! Thank you! I didn’t find FLINK-18712 and the corresponding PR. I read them and realized that jemalloc has already becamed default memory allocator of offical flink images since 1.12. Our flink image is built by ourself so we are still facing this memory leak issue. We w

[jira] [Created] (FLINK-33959) Unstable test case

2023-12-29 Thread RocMarshal (Jira)
RocMarshal created FLINK-33959: -- Summary: Unstable test case Key: FLINK-33959 URL: https://issues.apache.org/jira/browse/FLINK-33959 Project: Flink Issue Type: Bug Components: Tests

Re: Anyone encounters the non-heap memory increasing issue when using RocksDB as state backend and has better solutions?

2023-12-29 Thread Zakelly Lan
Hi Sue Alen, AFAIK, Flink encountered memory leak issues with RocksDB block cache. The root cause is the memory fragmentation with glibc. You may get some information in FLINK-18712[1]. Actually there are some efforts from the Flink side, such as using jemalloc as default in image[2]. I have no id