Hi Li, Regarding your questions:
1. Is there actually a way to pass in the heap size via arguments to > taskmanager.sh? Is passing -Dtaskmanager.heap.size supposed to work? No, '-Dtaskmanager.heap.size' is not supposed to work. The '-D' configurations are only parsed after the JVM is started, while 'taskmanager.heap.size' will be used for starting the JVM. 2. If not, is there a recommended way to set the heap size by environment, > like environmental variables? Yes, these is an equivalent environment variable FLINK_TM_HEAP. 3. Also, the maximum heap size logged and -Xms and -Xmx is always a little > smaller than the configured size (i.e. configuring 3000m results 2700m in > the jvm arguments, 1024m results in 922), why is that? The configuration key 'taskmanager.heap.size' is a bit inaccurate. The config option actually also account for some off-heap memory, such as network direct buffers and off-heap managed memory (if used). That's way you see the java heap size is always slightly smaller than the configured 'taskmanager.heap.size'. Thank you~ Xintong Song On Wed, Jan 1, 2020 at 3:10 AM Li Peng <li.p...@doordash.com> wrote: > Hey folks, we've been running a k8 flink application, using the > taskmanager.sh script and passing in the -Djobmanager.heap.size=9000m and > -Dtaskmanager.heap.size=7000m as options to the script. I noticed from the > logs, that the Maximum heap size logged completely ignores these arguments, > and just sets the heap to the default of 922M. > > I tested setting the taskmanager.heap.size and jobmanager.heap.size > manually in flink-conf.yaml, and it does work as expected (minus the heap > being set a little lower than configured). But since we want the > application to pick up different memory settings based on the environment > (local/staging/prod/etc), setting it in flink-conf isn't ideal. > > So my questions are: > > 1. Is there actually a way to pass in the heap size via arguments to > taskmanager.sh? Is passing -Dtaskmanager.heap.size supposed to work? > 2. If not, is there a recommended way to set the heap size by > environment, like environmental variables? > 3. Also, the maximum heap size logged and -Xms and -Xmx is always a little > smaller than the configured size (i.e. configuring 3000m results 2700m in > the jvm arguments, 1024m results in 922), why is that? > > Thanks, and happy new year! > Li > >
