+1 for the FLIP. I like this FLIP. It is very useful.
I'd like to ask one more question:
"
- The internal parameters passed in by the user contains the external
parameter. At this point, we can see that the user configures the same two
parameters in the *external parameter* and *internal parameters*, in
this case the Outer parameters is valid ("allowNonRestoredState": true)
"
Should we enable one more feature to let users decide which parameter will
win if there is a conflict between the external parameter and internal
parameter? By default, like you described, the external parameter will win.
Beyond that, users could pass an additional parameter to tell Flink that
all parameters in flinkConfiguration have higher priorities and will
overwrite the value of the external parameter with the same name.
This additional parameter e.g. "overwriteExternalParameter" (more feasible
name could be reconsidered) could be defined within the flinkConfiguration
like:
{
"entryClass":
"org.apache.flink.streaming.examples.statemachine.StateMachineExample",
"parallelism": "1",
"programArgs": null,
"savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
"allowNonRestoredState": true,
"flinkConfiguration": {
*"overwriteExternalParameter" : true, // default value is false for
backward compatibility*
"allowNonRestoredState": false,
"state.checkpoints.dir":
"hdfs://flink/jobs/stateMachineExample/checkpoints"
}
}
Or as a new external parameter e.g. "overwriteFlinkConfiguration"
{
"entryClass":
"org.apache.flink.streaming.examples.statemachine.StateMachineExample",
"parallelism": "1",
"programArgs": null,
"savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
"allowNonRestoredState": true,
*"overwriteFlinkConfiguration" : false, // default value is true for
backward compatibility*
"flinkConfiguration": {
"allowNonRestoredState": false,
"state.checkpoints.dir":
"hdfs://flink/jobs/stateMachineExample/checkpoints"
}
}
In this way, we will give even more flexibility to users. WDYT?
Best regards,
Jing
On Thu, Aug 18, 2022 at 12:52 PM Teoh, Hong <lian...@amazon.co.uk.invalid>
wrote:
> @Gyula: I was thinking of the use case where the managed platform also
> handles the job submission.
>
> Setup #1
> External user : Creating jar
> ------------------------------------
> Managed platform : Submit job
> Managed platform : Manage Flink cluster
>
> As compared to
>
> Setup #2
> External user : Creating jar
> External user : Submit job
> ------------------------------------
> Managed platform : Manage Flink cluster
>
>
> But I take your point that a REST API toggle doesn't make sense at all in
> Setup #2, so it would be more inclusive to include the cluster
> configuration since that works for both setups
>
>
> Regards,
> Hong
>
>
>
> On 18/08/2022, 11:38, "Gyula Fóra" <gyula.f...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> +1 for the proposal.
>
> @Hong : I feel that the inverted override flag should be a cluster
> setting
> and not something the user can override at will. I fear that this might
> defeat the purpose of the feature itself.
> So I think we should add a config for this but not expose it on the
> rest api
>
> Gyula
>
> On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> <lian...@amazon.co.uk.invalid>
> wrote:
>
> > +1 to this FLIP.
> > This is very useful for teams building a Flink platform to run jobs
> from
> > an external user
> >
> > +1 on Danny's comment on adding a configuration to allow inverted
> order of
> > overrides.
> > However, might it be better to include an "override" toggle in the
> REST
> > API itself? That way we can change the flink configuration override
> > behaviour without restarting the Flink cluster. This would make
> sense if we
> > are thinking of a Session cluster and deploying multiple Flink jobs
> to the
> > same cluster.
> >
> > Regards,
> > Hong
> >
> >
> > On 18/08/2022, 10:46, "Danny Cranmer" <dannycran...@apache.org>
> wrote:
> >
> > CAUTION: This email originated from outside of the organization.
> Do
> > not click links or open attachments unless you can confirm the
> sender and
> > know the content is safe.
> >
> >
> >
> > +1 thanks for driving this FLIP.
> >
> > We actually have an internally forked equivalent of this, which
> is on
> > our
> > list to try to upstream. Your proposal would *almost* work "off
> the
> > shelf"
> > for us. We have an inverted order or priority:
> > - Rest API / Flink CLI > User Code > Cluster Config
> >
> > This is because our end users do not submit their jobs directly,
> our
> > service does it on their behalf. For our usecase we do not want
> to
> > allow
> > users to override certain values we set, since some are managed
> from
> > our
> > service configuration, example: checkpointing interval.
> >
> > How would you feel about including a cluster configuration to
> invert
> > the
> > order? This could be decoupled to a follow-up FLIP.
> >
> > Thanks,
> > Danny Cranmer
> >
> >
> > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> jam.gz...@gmail.com>
> > wrote:
> >
> > > This is a good suggestion, we can start this work after we
> finish the
> > > discussion and vote
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> > >
> > > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道：
> > >
> > > > I've created a new ticket [FLINK-28973] Extending
> > /jars/:jarid/plan API
> > > to
> > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the
> job
> > plan
> > > API.
> > > > Maybe we can create a new umbrella ticket for FLIP-256 and
> put
> > > FLINK-27060
> > > > & FLINK-28973 as subtasks. WDYT?
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: zhengyu chen <jam.gz...@gmail.com>
> > > > Sent: Monday, August 15, 2022 18:06
> > > > To: dev@flink.apache.org <dev@flink.apache.org>
> > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter
> With
> > Flink Rest
> > > > Api
> > > >
> > > > Hi all,
> > > >
> > > >
> > > > We would like to start a discussion thread on FLIP-256
> Support Job
> > > Dynamic
> > > > Parameter With Flink Rest Api
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > [1]
> > > >
> > > > After the user submits the jar package, running a job through
> > restapi
> > > > (/jars/:jarid/run) [2] can only pass in
> (allowNonRestoredState,
> > > > savepointPath, programArg, entry-class, parallelism)
> parameters,
> > which is
> > > > obvious with the diversification of job parameters (eg
> Checkpoint
> > > address)
> > > >
> > > > This solves the problem that the user can pass in other
> parameters
> > when
> > > > submitting a job, avoiding the user to define these job
> parameters
> > in the
> > > > code, resulting in the need to repackage the job for each
> > modification
> > > >
> > > > There was some interest from users [3] from a meetup and the
> > mailing
> > > list.
> > > > Looking forward to comments and feedback, thanks!
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > >
> > > >
> > > > [2]
> > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > >
> > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > >
> >
> >
>
>
