HI Hong, Maybe you forgot. I don’t know if the current FLIP still needs to be modified. If not, we will start with the current plan. If you have relevant ideas in the future, please continue to discuss. If not, we will open voting at a later date.
Teoh, Hong <lian...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道： > > Hi Zheng Yu, > > We would have to take the same "cluster configuration" (cannot be set on job > submission) vs "job configuration" approach in the User code as well. And we > can classify jobmanager.rest.api.submit.job.allow-reset-config as a cluster > configuration. That way, neither the REST API / User code can override this > configuration, and it can only be set in cluster configuration on startup. > > There are other cluster specific configurations that are already treated this > way (e.g. jobmanager.rpc.address, jobmanager.memory.process.size). As part of > this work, I wonder if we can update the Flink docs on configuration > (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/) > to specify which configuration is "cluster" and which is "job". > > Regards, > Hong > > > > On 20/08/2022, 09:16, "Zheng Yu Chen" <jam.gz...@gmail.com> wrote: > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > @Gyula say that add a config for this but not expose it on the rest > api is right > when user start up session cluster,we can config (eg > jobmanager.rest.api.submit.job.allow-reset-config = true/false) to > controller this behavior > > Now we have an existing cluster, the admin configures checkpointing > interval by this session cluster > > jobmanager.rest.api.submit.job.allow-reset-config = true/false > > - true : allow user reset new value config > - false(default) : not allow user set new config from ,throw exp to > tell user what properties is admin was set it > > However, there will be a problem in doing this. If the user writes > this Flink Config in jar by hardcoding, the highest priority must be > the code at this time, so the problem may still exist, and the user > cannot be prevented from this behavior. > > what do you think ? > > @Hong @Gyula @Teoh @Danny > > > Gyula Fóra <gyula.f...@gmail.com> 于2022年8月18日周四 18:37写道： > > > > +1 for the proposal. > > > > @Hong : I feel that the inverted override flag should be a cluster > setting > > and not something the user can override at will. I fear that this might > > defeat the purpose of the feature itself. > > So I think we should add a config for this but not expose it on the > rest api > > > > Gyula > > > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong > <lian...@amazon.co.uk.invalid> > > wrote: > > > > > +1 to this FLIP. > > > This is very useful for teams building a Flink platform to run jobs > from > > > an external user > > > > > > +1 on Danny's comment on adding a configuration to allow inverted > order of > > > overrides. > > > However, might it be better to include an "override" toggle in the > REST > > > API itself? That way we can change the flink configuration override > > > behaviour without restarting the Flink cluster. This would make sense > if we > > > are thinking of a Session cluster and deploying multiple Flink jobs > to the > > > same cluster. > > > > > > Regards, > > > Hong > > > > > > > > > On 18/08/2022, 10:46, "Danny Cranmer" <dannycran...@apache.org> wrote: > > > > > > CAUTION: This email originated from outside of the organization. > Do > > > not click links or open attachments unless you can confirm the sender > and > > > know the content is safe. > > > > > > > > > > > > +1 thanks for driving this FLIP. > > > > > > We actually have an internally forked equivalent of this, which > is on > > > our > > > list to try to upstream. Your proposal would *almost* work "off > the > > > shelf" > > > for us. We have an inverted order or priority: > > > - Rest API / Flink CLI > User Code > Cluster Config > > > > > > This is because our end users do not submit their jobs directly, > our > > > service does it on their behalf. For our usecase we do not want to > > > allow > > > users to override certain values we set, since some are managed > from > > > our > > > service configuration, example: checkpointing interval. > > > > > > How would you feel about including a cluster configuration to > invert > > > the > > > order? This could be decoupled to a follow-up FLIP. > > > > > > Thanks, > > > Danny Cranmer > > > > > > > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen > <jam.gz...@gmail.com> > > > wrote: > > > > > > > This is a good suggestion, we can start this work after we > finish the > > > > discussion and vote > > > > > > > > -- > > > > Best > > > > > > > > ConradJam > > > > > > > > > > > > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道： > > > > > > > > > I've created a new ticket [FLINK-28973] Extending > > > /jars/:jarid/plan API > > > > to > > > > > support setting Flink configs - ASF JIRA (apache.org)< > > > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the > job > > > plan > > > > API. > > > > > Maybe we can create a new umbrella ticket for FLIP-256 and put > > > > FLINK-27060 > > > > > & FLINK-28973 as subtasks. WDYT? > > > > > > > > > > Best, > > > > > Zhanghao Chen > > > > > ________________________________ > > > > > From: zhengyu chen <jam.gz...@gmail.com> > > > > > Sent: Monday, August 15, 2022 18:06 > > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With > > > Flink Rest > > > > > Api > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > We would like to start a discussion thread on FLIP-256 > Support Job > > > > Dynamic > > > > > Parameter With Flink Rest Api > > > > > < > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > > > > > [1] > > > > > > > > > > After the user submits the jar package, running a job through > > > restapi > > > > > (/jars/:jarid/run) [2] can only pass in > (allowNonRestoredState, > > > > > savepointPath, programArg, entry-class, parallelism) > parameters, > > > which is > > > > > obvious with the diversification of job parameters (eg > Checkpoint > > > > address) > > > > > > > > > > This solves the problem that the user can pass in other > parameters > > > when > > > > > submitting a job, avoiding the user to define these job > parameters > > > in the > > > > > code, resulting in the need to repackage the job for each > > > modification > > > > > > > > > > There was some interest from users [3] from a meetup and the > > > mailing > > > > list. > > > > > Looking forward to comments and feedback, thanks! > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run > > > > > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060 > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best > > > > > > > > > > ConradJam > > > > > > > > > > > > > > > > -- Best ConradJam
