Hi all,
Thanks for opening the discussion!
I like the idea, so +1 from my side and actually this is aligned with
our intensions for the FLIP-73 effort.
For the naming convention of the parameters introduced in the FLIP, my
proposal would be have the full word "execution" instead of the
shorter "exec".
The reason for this, is that in the context of FLIP-73, we are also
planning to introduce some new configuration parameters and the
convention we
are currently using is the following:
pipeline.***: for job parameters that will not change between
executions of the same job, e.g. the jar location
executor.***: for parameters relevant to the instantiation of the
correct executor, e.g. YARN, detached, etc
execution.***: for parameters that are relevant to a specific
execution of a given pipeline, e.g. parallelism or savepoint settings
I understand that sometimes the boundaries may not be that clear for a
parameter but I hope this will not be relevant to most of the
parameters.
I will also open a FLIP with some addition parameters but until then,
this is the scheme that we are planning to follow.
Cheers,
Kostas
On Mon, Sep 2, 2019 at 9:26 AM Dawid Wysakowicz <dwysakow...@apache.org> wrote:
>
> Hi Gyula,
>
> Yes you are right, we were also considering the external configurer. The
> reason we suggest the built in method is that it is more tightly coupled
> with the place the options are actually set. Therefore our hope is that,
> whenever somebody e.g. adds new fields to the ExecutionConfig he/she
> updates also the configure method. I am not entirely against your
> suggestion though, if this is the preferred way in the community.
>
> Does anyone has any comments regarding the option keys?
>
> Best,
>
> Dawid
>
> On 30/08/2019 14:57, Gyula Fóra wrote:
> > Hi Dawid,
> >
> > Sorry I misread one of the interfaces a little (Configuration instead of
> > ConfigurationReader), you are right.
> > I was referring to:
> >
> >
> > -
> >
> > void StreamExecutionEnvironment.configure(ConfigurationReader)
> >
> >
> > This might be slightly orthogonal to the changes that you made here but
> > what I meant is that instead of adding methods to the
> > StreamExecutionEnvironment we could make this an external interface:
> >
> > EnvironmentConfigurer {
> > void configure(StreamExecutionEnvironment, ConfigurationReader)
> > }
> >
> > We could then have a default implementation of the EnvironmentConfigurer
> > that would understand built in options. We could also allow users to pass
> > custom implementations of this, which could configure the
> > StreamExecutionEnvironment based on user defined config options. This is
> > just a rough idea for extensibility and probably out of scope at first.
> >
> > Cheers,
> > Gyula
> >
> > On Fri, Aug 30, 2019 at 12:13 PM Dawid Wysakowicz <dwysakow...@apache.org>
> > wrote:
> >
> >> Hi Gyula,
> >>
> >> Thank you for the support on those changes.
> >>
> >> I am not sure if I understood your idea for the "reconfiguration" logic.
> >>
> >> The configure method on those objects would take ConfigurationReader. So
> >> user can provide a thin wrapper around Configuration for e.g. filtering
> >> certain logic, changing values based on other parameters etc. Is that
> >> what you had in mind?
> >>
> >> Best,
> >>
> >> Dawid
> >>
> >> On 29/08/2019 19:21, Gyula Fóra wrote:
> >>> Hi!
> >>>
> >>> Huuuge +1 from me, this has been an operational pain for years.
> >>> This would also introduce a nice and simple way to extend it in the
> >> future
> >>> if we need.
> >>>
> >>> Ship it!
> >>>
> >>> Gyula
> >>>
> >>> On Thu, Aug 29, 2019 at 5:05 PM Dawid Wysakowicz <dwysakow...@apache.org
> >>>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I wanted to propose a new, additional way of configuring execution
> >>>> parameters that can currently be set only on such objects like
> >>>> ExecutionConfig, CheckpointConfig and StreamExecutionEnvironment. This
> >>>> poses problems such as:
> >>>>
> >>>> - no easy way to configure those from a file
> >>>> - there is no easy way to pass a configuration from layers built on
> >>>> top of StreamExecutionEnvironment. (e.g. when we want to configure
> >> those
> >>>> options from TableEnvironment)
> >>>> - they are not automatically documented
> >>>>
> >>>> Note that there are a few concepts from FLIP-54[1] that this FLIP is
> >> based
> >>>> on.
> >>>>
> >>>> Would be really grateful to know if you think this would be a valuable
> >>>> addition and any other feedback.
> >>>>
> >>>> Best,
> >>>>
> >>>> Dawid
> >>>>
> >>>> Wiki page:
> >>>>
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object
> >>>> Google doc:
> >>>>
> >> https://docs.google.com/document/d/1l8jW2NjhwHH1mVPbLvFolnL2vNvf4buUMDZWMfN_hFM/edit?usp=sharing
> >>>>
> >>>> [1]
> >>>>
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration
> >>>>
> >>>>
> >>
>
