Thanks all for the discussion.
We agreed to have process.size in the default settings with the explanation
> of flink.size alternative in the comment.
> This way we keep -tm as a shortcut to process.size only and any
> inconsistencies fail fast as if configured in yaml.
>
The conclusions sounds g
Hi all,
Stephan, Till and me had another offline discussion today. Here is the
outcome of our brainstorm.
We agreed to have process.size in the default settings with the explanation
of flink.size alternative in the comment.
This way we keep -tm as a shortcut to process.size only and any
inconsist
I think that problem exists anyways and is independent of the "-tm" option.
You can have a combination of `task.heap.size` and `managed.size` and
`framework.heap.size` that conflicts with `flink.size`. In that case, we
get an exception during the startup (incompatible memory configuration)?
That i
Clearing the `flink.size` option and setting `process.size` could indeed be
a solution. The thing I'm wondering is what would happen if the user has
configured `task.heap.size` and `managed.size` instead of `flink.size`?
Would we also ignore these settings? If not, then we risk to run into the
situ
True, even we have "process.size" rather than "flink.size" in the default
config file, user can still have "flink.size" in their custom config file.
If we consider "-tm" as a shortcut for users to override the TM memory
size, then it makes less sense to require users to remove "flink.size" from
the
Would be good to hear the thoughts of some more Yarn users, though.
On Mon, Jan 13, 2020 at 7:23 PM Stephan Ewen wrote:
> I think we need an interpretation of "-tm" regardless of what is in the
> default configuration, because we can always have a modified configuration
> and then a user passes
I think we need an interpretation of "-tm" regardless of what is in the
default configuration, because we can always have a modified configuration
and then a user passes the "-tm" flag.
I kind of like the first proposal: Interpret "-tm" as "override memory size
config and set the Yarn TM container
Hi all,
While working on changing process memory to Flink memory in default
configuration, Xintong encountered a problem.
When -tm option is used to rewrite container memory, basically process
memory, it can collide with the default Flink memory.
For legacy users it should not be a problem as we a
Great! Thanks, guys, for the continued effort on this topic!
On Thu, Jan 9, 2020 at 5:27 AM Xintong Song wrote:
> Thanks all for the discussion. I believe we have get consensus on all the
> open questions discussed in this thread.
>
> Since Andrey already create a jira ticket for renaming shuffl
Thanks all for the discussion. I believe we have get consensus on all the
open questions discussed in this thread.
Since Andrey already create a jira ticket for renaming shuffle memory
config keys with "taskmanager.memory.network.*", I'll create ticket for the
other topic that puts flink.size in f
It also looks to me that we should only swap network and memory in the
option names: 'taskmanager.memory.network.*'.
There is no strong consensus towards using new 'shuffle' naming so we can
just rename it to 'taskmanager.memory.network.*' as 'shuffle' naming has
never been released.
When we have
About the name for the key "taskmanager.memory.network.*" or
"taskmanager.memory.shuffle.*":
This is a nice to have, but not super critical. If there ends up being no
consensus, I can withdraw my concern.
On Wed, Jan 8, 2020 at 4:01 AM Xintong Song wrote:
> Thanks for the explanations @Stephan,
Thanks for the explanations @Stephan, and feedbacks @Jingsong @Andrey
@Till. I'm also ok with flink.size in flink-conf.yaml.
And if I understand correctly, we have also get consensus on having the
configuration key 'taskmanager.memory.network.*' (with deprecated key
'taskmanager.network.memory.*')
I'm also slightly in favour of the flink.size instead of process.size. If
Xintong is ok with this, then we should update the flink-conf.yaml in the
repo.
Cheers,
Till
On Mon, Jan 6, 2020 at 4:09 PM Andrey Zagrebin wrote:
> Thank you for more explanation Stephan and feedback Jingsong,
>
> I see
Thank you for more explanation Stephan and feedback Jingsong,
I see the point now.
‘taskmanager.memory.flink.size’ is indeed simpler to understand for the
newcomers in the default config because it includes less types of memory to
consider.
Most of the time beginners will think about heap size and
Thank you for your wonderful discussion.
+1 for set "taskmanager.memory.flink.size" in the default config.
Maybe we can write and explain "taskmanager.memory.process.size" in the
comments.
Most of the time,
- "trying out" users is less in-depth users and use standalone mode.
- Production users us
"taskmanager.memory.flink.size" in the default config has a few advantages.
- The value in the default config needs to be suitable for "trying out"
Flink, for a good "getting started" experience.
- For trying out Flink, standalone is the most common entry point (except
running in IDE).
- In
Thanks for the summary, Xintong! It makes sense to me.
How about putting "taskmanager.memory.flink.size" in the configuration?
> Then new downloaded Flink behaves similar to the previous Standalone setups.
> If someone upgrades the binaries, but re-uses their old configuration,
> then they get the
Hi all,
It seems we have already get consensus on most of the issues. Thanks
everyone for the good discussion.
While there are still open questions under discussion, I'd like to
summarize the discussion so far, and list the action items that we already
get consensus on. In this way, we can alread
>
> How about putting "taskmanager.memory.flink.size" in the configuration?
> Then new downloaded Flink behaves similar to the previous Standalone setups.
> If someone upgrades the binaries, but re-uses their old configuration,
> then they get the compatibility as discussed previously.
> We used th
>
> How about putting "taskmanager.memory.flink.size" in the configuration?
> Then new downloaded Flink behaves similar to the previous Standalone setups.
> If someone upgrades the binaries, but re-uses their old configuration,
> then they get the compatibility as discussed previously.
> We used th
How about putting "taskmanager.memory.flink.size" in the configuration?
Then new downloaded Flink behaves similar to the previous Standalone setups.
If someone upgrades the binaries, but re-uses their old configuration, then
they get the compatibility as discussed previously.
We used that approach
>
> +1 to not have more options for off-heap memory, that would get confusing
> fast. We can keep the name "off-heap" for both task and framework, as long
> as they mean the same thing: native plus direct, and fully counted into
> MaxDirectMemory. I would suggest to update the config descriptions t
Thanks for the feedback and good discussion everyone. I left some comments
inline.
On Fri, Dec 20, 2019 at 1:59 PM Stephan Ewen wrote:
> +1 to not have more options for off-heap memory, that would get confusing
> fast. We can keep the name "off-heap" for both task and framework, as long
> as the
+1 to not have more options for off-heap memory, that would get confusing
fast. We can keep the name "off-heap" for both task and framework, as long
as they mean the same thing: native plus direct, and fully counted into
MaxDirectMemory. I would suggest to update the config descriptions then to
ref
Regarding the framework/task direct/native memory options, I tend to think
it differently. I'm in favor of keep the "*.off-heap.size" for the config
option keys.
- It's not necessary IMO to expose the difference concepts of direct /
native memory to the users.
- I would avoid introducing
Hi Stephan and Xintong,
Thanks for the further FLIP-49 feedbacks.
> - "taskmanager.memory.size" (old main config option) is replaced by
> "taskmanager.memory.total-process.size" which has a different meaning in
> standalone setups. The old option did not subtract metaspace and other
> overhe
Hi Xintong!
Please find my answers inline:
> - "taskmanager.memory.size" (old main config option) is replaced by
>> "taskmanager.memory.total-process.size" which has a different meaning in
>> standalone setups. The old option did not subtract metaspace and other
>> overhead, while the new optio
Hi Stephan,
Thank you for kicking off this discussion and providing the suggestions.
- "taskmanager.memory.size" (old main config option) is replaced by
> "taskmanager.memory.total-process.size" which has a different meaning in
> standalone setups. The old option did not subtract metaspace and
Hi all!
I looked into using Flink with the new memory configurations and played
around a bit.
All in all, I think we made a really good step there in removing implicit
assumptions and rather making the configuration of memory sizes more
explicit. My experience was mostly very positive.
There were
30 matches
Mail list logo
