+1 to this FLIP.
This is very useful for teams building a Flink platform to run jobs from an
external user
+1 on Danny's comment on adding a configuration to allow inverted order of
overrides.
However, might it be better to include an "override" toggle in the REST API
itself? That way we can change the flink configuration override behaviour
without restarting the Flink cluster. This would make sense if we are thinking
of a Session cluster and deploying multiple Flink jobs to the same cluster.
Regards,
Hong
On 18/08/2022, 10:46, "Danny Cranmer" <dannycran...@apache.org> wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and know the
content is safe.
+1 thanks for driving this FLIP.
We actually have an internally forked equivalent of this, which is on our
list to try to upstream. Your proposal would *almost* work "off the shelf"
for us. We have an inverted order or priority:
- Rest API / Flink CLI > User Code > Cluster Config
This is because our end users do not submit their jobs directly, our
service does it on their behalf. For our usecase we do not want to allow
users to override certain values we set, since some are managed from our
service configuration, example: checkpointing interval.
How would you feel about including a cluster configuration to invert the
order? This could be decoupled to a follow-up FLIP.
Thanks,
Danny Cranmer
On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <jam.gz...@gmail.com> wrote:
> This is a good suggestion, we can start this work after we finish the
> discussion and vote
>
> --
> Best
>
> ConradJam
>
>
> <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道：
>
> > I've created a new ticket [FLINK-28973] Extending /jars/:jarid/plan API
> to
> > support setting Flink configs - ASF JIRA (apache.org)<
> > https://issues.apache.org/jira/browse/FLINK-28973>. for the job plan
> API.
> > Maybe we can create a new umbrella ticket for FLIP-256 and put
> FLINK-27060
> > & FLINK-28973 as subtasks. WDYT?
> >
> > Best,
> > Zhanghao Chen
> > ________________________________
> > From: zhengyu chen <jam.gz...@gmail.com>
> > Sent: Monday, August 15, 2022 18:06
> > To: dev@flink.apache.org <dev@flink.apache.org>
> > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink
Rest
> > Api
> >
> > Hi all,
> >
> >
> > We would like to start a discussion thread on FLIP-256 Support Job
> Dynamic
> > Parameter With Flink Rest Api
> > <
> >
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > >
> > [1]
> >
> > After the user submits the jar package, running a job through restapi
> > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > savepointPath, programArg, entry-class, parallelism) parameters, which
is
> > obvious with the diversification of job parameters (eg Checkpoint
> address)
> >
> > This solves the problem that the user can pass in other parameters when
> > submitting a job, avoiding the user to define these job parameters in
the
> > code, resulting in the need to repackage the job for each modification
> >
> > There was some interest from users [3] from a meetup and the mailing
> list.
> > Looking forward to comments and feedback, thanks!
> >
> >
> > [1]
> >
> >
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >
> >
> > [2]
> >
> >
>
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> >
> > [3] https://issues.apache.org/jira/browse/FLINK-27060
> >
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
