Hi folks,
I am setting up a Flink cluster for testing, and I have a few questions
regarding memory allocations:
1. Is there a recommended limit to the size of a TaskManager heap? I saw
that Flink uses G1GC, so we can use dozens of GB.
2. Following the above question, should I use only one TaskManager process
per machine, and give it all the available memory (minus a couple of GB for the
OS)?
3. Should I reserve some memory for RocksDB? The partitioned state will be
around 500GB in size, and to my understanding RocksDB runs in native code and
so uses off-heap memory.
4. What is the recommended heap size of a JobManager? I expect that the
cluster will run only 2 jobs at the same time.
The planned layout of the standalone cluster is:
* 3 small JobManager machines, running:
* 1 process of Zookeeper peer
* 1 JobManager process
* N large TaskManager machines, each running 1 TM process
Thanks!
Avihai
