Sounds good to me.
Thanks,
Timo
On 17.10.19 09:30, Kostas Kloudas wrote:
Hi Timo,
I agree that distinguishing between "executor" and "execution" when
scanning through a configuration file can be difficult. These names
were mainly influenced by the fact that FLIP-73 introduced the
"Executor".
In addition, I agree that "deployment" or "deploy" sound good
alternatives. Between the two, I would go with "deployment" (although
I like more the "deploy" as it is more imperative) for the simple
reason that we do not use verbs anywhere else (I think) in config
options.
Now for the "exec" or "execution", personally I like the longer
version as it is clearer.
So, to summarise, I would vote for "deployment", "execution", and
"pipeline" for job invariants, like the jars.
What do you think?
Cheers,
Kostas
On Wed, Oct 16, 2019 at 5:28 PM Timo Walther <twal...@apache.org> wrote:
Hi Kostas,
can we still discuss the naming of the properties? For me, having
"execution" and "exector" as prefixes might be confusing in the future
and difficult to identify if you scan through a list of properties.
How about `deployment` and `execution`? Or `deployer` and `exec`?
Regards,
Timo
On 16.10.19 16:31, Kostas Kloudas wrote:
Hi all,
Thanks for opening the discussion!
I like the idea, so +1 from my side and actually this is aligned with
our intensions for the FLIP-73 effort.
For the naming convention of the parameters introduced in the FLIP, my
proposal would be have the full word "execution" instead of the
shorter "exec".
The reason for this, is that in the context of FLIP-73, we are also
planning to introduce some new configuration parameters and the
convention we
are currently using is the following:
pipeline.***: for job parameters that will not change between
executions of the same job, e.g. the jar location
executor.***: for parameters relevant to the instantiation of the
correct executor, e.g. YARN, detached, etc
execution.***: for parameters that are relevant to a specific
execution of a given pipeline, e.g. parallelism or savepoint settings
I understand that sometimes the boundaries may not be that clear for a
parameter but I hope this will not be relevant to most of the
parameters.
I will also open a FLIP with some addition parameters but until then,
this is the scheme that we are planning to follow.
Cheers,
Kostas
On Mon, Sep 2, 2019 at 9:26 AM Dawid Wysakowicz <dwysakow...@apache.org> wrote:
Hi Gyula,
Yes you are right, we were also considering the external configurer. The
reason we suggest the built in method is that it is more tightly coupled
with the place the options are actually set. Therefore our hope is that,
whenever somebody e.g. adds new fields to the ExecutionConfig he/she
updates also the configure method. I am not entirely against your
suggestion though, if this is the preferred way in the community.
Does anyone has any comments regarding the option keys?
Best,
Dawid
On 30/08/2019 14:57, Gyula Fóra wrote:
Hi Dawid,
Sorry I misread one of the interfaces a little (Configuration instead of
ConfigurationReader), you are right.
I was referring to:
-
void StreamExecutionEnvironment.configure(ConfigurationReader)
This might be slightly orthogonal to the changes that you made here but
what I meant is that instead of adding methods to the
StreamExecutionEnvironment we could make this an external interface:
EnvironmentConfigurer {
void configure(StreamExecutionEnvironment, ConfigurationReader)
}
We could then have a default implementation of the EnvironmentConfigurer
that would understand built in options. We could also allow users to pass
custom implementations of this, which could configure the
StreamExecutionEnvironment based on user defined config options. This is
just a rough idea for extensibility and probably out of scope at first.
Cheers,
Gyula
On Fri, Aug 30, 2019 at 12:13 PM Dawid Wysakowicz <dwysakow...@apache.org>
wrote:
Hi Gyula,
Thank you for the support on those changes.
I am not sure if I understood your idea for the "reconfiguration" logic.
The configure method on those objects would take ConfigurationReader. So
user can provide a thin wrapper around Configuration for e.g. filtering
certain logic, changing values based on other parameters etc. Is that
what you had in mind?
Best,
Dawid
On 29/08/2019 19:21, Gyula Fóra wrote:
Hi!
Huuuge +1 from me, this has been an operational pain for years.
This would also introduce a nice and simple way to extend it in the
future
if we need.
Ship it!
Gyula
On Thu, Aug 29, 2019 at 5:05 PM Dawid Wysakowicz <dwysakow...@apache.org
wrote:
Hi,
I wanted to propose a new, additional way of configuring execution
parameters that can currently be set only on such objects like
ExecutionConfig, CheckpointConfig and StreamExecutionEnvironment. This
poses problems such as:
- no easy way to configure those from a file
- there is no easy way to pass a configuration from layers built on
top of StreamExecutionEnvironment. (e.g. when we want to configure
those
options from TableEnvironment)
- they are not automatically documented
Note that there are a few concepts from FLIP-54[1] that this FLIP is
based
on.
Would be really grateful to know if you think this would be a valuable
addition and any other feedback.
Best,
Dawid
Wiki page:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object
Google doc:
https://docs.google.com/document/d/1l8jW2NjhwHH1mVPbLvFolnL2vNvf4buUMDZWMfN_hFM/edit?usp=sharing
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration
