Hi Lasse Would you please give more details?
1. What is the memory configuration of your task manager? e.g the memory size of process, managed memory. And how large the memory would increase to once you meet problem. 2. Did you use managed memory to control RocksDB? [1] 3. Why you give the assumption that memory problem has relationship with savepoint? 4. What is the topology of you streaming job: how many states and window you use, how many slots per task manager? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_tuning.html#rocksdb-state-backend Best Yun Tang ________________________________ From: Lasse Nedergaard <lassenedergaardfl...@gmail.com> Sent: Thursday, April 30, 2020 12:39 To: Yun Tang <myas...@live.com> Cc: user <user@flink.apache.org> Subject: Re: Savepoint memory overhead We using Flink 1.10 running on Mesos. Med venlig hilsen / Best regards Lasse Nedergaard Den 30. apr. 2020 kl. 04.53 skrev Yun Tang <myas...@live.com>: Hi Lasse Which version of Flink did you use? Before Flink-1.10, there might exist memory problem when RocksDB executes savepoint with write batch[1]. [1] https://issues.apache.org/jira/browse/FLINK-12785 Best Yun Tang ________________________________ From: Lasse Nedergaard <lassenedergaardfl...@gmail.com> Sent: Wednesday, April 29, 2020 21:17 To: user <user@flink.apache.org> Subject: Savepoint memory overhead Hi. I would like to know if there are any guidelines/recommendations for the memory overhead we need to calculate for when doing savepoint to s3. We use RockDb state backend. We run our job on relative small task managers and we can see we get memory problems if the state size for each task manager get "big" (we haven't found the rule of thumbs yet) and we can remove the problem if we reduce the state size, or increase parallelism and jobs with none or small state don't have any problems. So I see a relation between between allocated memory to a task manager and the state it can handle. So do anyone have any recommendations/ base practices for this and can someone explain why savepoint requires memory. Thanks In advance Lasse Nedergaard
