Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10

2020-04-30 Thread Jiahui Jiang
I see I see. Thank you so much! From: Xintong Song Sent: Wednesday, April 29, 2020 11:22 PM To: Jiahui Jiang Cc: user@flink.apache.org Subject: Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10 That's pretty much it. I'm not very fam

Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10

2020-04-29 Thread Xintong Song
them, we need to set up a non-zero task.off-heap? > > Thanks! > -- > *From:* Xintong Song > *Sent:* Wednesday, April 29, 2020 10:53 PM > *To:* Jiahui Jiang > *Cc:* Steven Wu ; user@flink.apache.org < > user@flink.apache.org> > *Subj

Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10

2020-04-29 Thread Jiahui Jiang
usage within them, we need to set up a non-zero task.off-heap? Thanks! From: Xintong Song Sent: Wednesday, April 29, 2020 10:53 PM To: Jiahui Jiang Cc: Steven Wu ; user@flink.apache.org Subject: Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.1

Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10

2020-04-29 Thread Xintong Song
to do! > > I will test our some more pipeline with different resource requirements to > understand the memory profiles better! Thank you guys again! > > -- > *From:* Steven Wu > *Sent:* Wednesday, April 29, 2020 10:12 AM > *To:* Xintong Song

Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10

2020-04-29 Thread Jiahui Jiang
tong Song Cc: Jiahui Jiang ; user@flink.apache.org Subject: Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10 Jiahui, Based on my reading on the doc, for containerized environment, it is probably better to set `taskmanager.memory.process.size` to the container memory limit.

Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10

2020-04-29 Thread Steven Wu
Jiahui, Based on my reading on the doc, for containerized environment, it is probably better to set `taskmanager.memory.process.size` to the container memory limit. https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#taskmanager-memory-process-size Then I typically set `taskman

Re: Configuring taskmanager.memory.task.off-heap.size in Flink 1.10

2020-04-28 Thread Xintong Song
Hi Jiahui, 'taskmanager.memory.task.off-heap.size' accounts for the off-heap memory reserved for your job / operators. There are other configuration options accounting for the off-heap memory usages for other purposes, e.g., 'taskmanager.memory.framework.off-heap'. The default 'task.off-heap.size'