Originally I assumed that user code programmatic config changes always have highest priority in Flink. If I understand correctly this is true in other deployment modes .
So I wouldn’t change that with a flag, the question is whether cluster config or rest api config has a higher priority. I think the default could be that rest api has a higher priority so: (Default) override true: User code > rest api > cluster conf Override false : User code > cluster conf > rest api I might be wrong about the current behavior though… Gyula On Sun, 28 Aug 2022 at 07:45, Zheng Yu Chen <jam.gz...@gmail.com> wrote: > Hi Hong: > > Well, I think this should be the same idea as Gyula said before > @Gyula What do you think ? > > If it is ok I will add this part to the documentation,Then the next process > should be initiated by the community to vote this filp, I don't know who > will host this part? Can someone help me with this 😀 > > > Teoh, Hong <lian...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道： > > > Hi Zheng Yu, > > > > Sorry for the late reply, I was on holiday last week. > > > > Could we propose the following config instead? > > > > rest.submit.job.override-config = true/false > > - true: REST API will have priority over user code (i.e. Rest API / > > Flink CLI > User Code > Cluster Config) > > - false (default): user code will have priority over REST API (i.e. > User > > Code > Rest API / Flink CLI > Cluster Config) > > > > This way, the default behaviour will be according to the proposed FLIP, > > but we will have an additional toggle to ignore the configuration set in > > user code. > > > > > However, there will be a problem in doing this. If the user > writes > > > this Flink Config in jar by hardcoding, the highest priority > > must be > > > the code at this time, so the problem may still exist, and the > > user > > > cannot be prevented from this behavior > > > > Hmm, would it be better if we set it such that the > > "rest.submit.job.override-config" cannot be overwritten in user code > > (ignored), just like configuration "jobmanager.rpc.port"? > > > > What do you think? > > > > Regards, > > Hong > > > > > > > > On 26/08/2022, 12:05, "Zheng Yu Chen" <jam.gz...@gmail.com> wrote: > > > > CAUTION: This email originated from outside of the organization. Do > > not click links or open attachments unless you can confirm the sender and > > know the content is safe. > > > > > > > > HI Hong, > > > > Maybe you forgot. I don’t know if the current FLIP still needs to be > > modified. If not, we will start with the current plan. If you have > > relevant ideas in the future, please continue to discuss. If not, we > > will open voting at a later date. > > > > Teoh, Hong <lian...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道： > > > > > > Hi Zheng Yu, > > > > > > We would have to take the same "cluster configuration" (cannot be > > set on job submission) vs "job configuration" approach in the User code > as > > well. And we can classify > jobmanager.rest.api.submit.job.allow-reset-config > > as a cluster configuration. That way, neither the REST API / User code > can > > override this configuration, and it can only be set in cluster > > configuration on startup. > > > > > > There are other cluster specific configurations that are already > > treated this way (e.g. jobmanager.rpc.address, > > jobmanager.memory.process.size). As part of this work, I wonder if we can > > update the Flink docs on configuration ( > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ > ) > > to specify which configuration is "cluster" and which is "job". > > > > > > Regards, > > > Hong > > > > > > > > > > > > On 20/08/2022, 09:16, "Zheng Yu Chen" <jam.gz...@gmail.com> wrote: > > > > > > CAUTION: This email originated from outside of the > organization. > > Do not click links or open attachments unless you can confirm the sender > > and know the content is safe. > > > > > > > > > > > > @Gyula say that add a config for this but not expose it on > the > > rest > > > api is right > > > when user start up session cluster,we can config (eg > > > jobmanager.rest.api.submit.job.allow-reset-config = true/false) > > to > > > controller this behavior > > > > > > Now we have an existing cluster, the admin configures > > checkpointing > > > interval by this session cluster > > > > > > jobmanager.rest.api.submit.job.allow-reset-config = true/false > > > > > > - true : allow user reset new value config > > > - false(default) : not allow user set new config from ,throw > exp > > to > > > tell user what properties is admin was set it > > > > > > However, there will be a problem in doing this. If the user > > writes > > > this Flink Config in jar by hardcoding, the highest priority > > must be > > > the code at this time, so the problem may still exist, and the > > user > > > cannot be prevented from this behavior. > > > > > > what do you think ? > > > > > > @Hong @Gyula @Teoh @Danny > > > > > > > > > Gyula Fóra <gyula.f...@gmail.com> 于2022年8月18日周四 18:37写道： > > > > > > > > +1 for the proposal. > > > > > > > > @Hong : I feel that the inverted override flag should be a > > cluster setting > > > > and not something the user can override at will. I fear that > > this might > > > > defeat the purpose of the feature itself. > > > > So I think we should add a config for this but not expose it > > on the rest api > > > > > > > > Gyula > > > > > > > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong > > <lian...@amazon.co.uk.invalid> > > > > wrote: > > > > > > > > > +1 to this FLIP. > > > > > This is very useful for teams building a Flink platform to > > run jobs from > > > > > an external user > > > > > > > > > > +1 on Danny's comment on adding a configuration to allow > > inverted order of > > > > > overrides. > > > > > However, might it be better to include an "override" toggle > > in the REST > > > > > API itself? That way we can change the flink configuration > > override > > > > > behaviour without restarting the Flink cluster. This would > > make sense if we > > > > > are thinking of a Session cluster and deploying multiple > > Flink jobs to the > > > > > same cluster. > > > > > > > > > > Regards, > > > > > Hong > > > > > > > > > > > > > > > On 18/08/2022, 10:46, "Danny Cranmer" < > > dannycran...@apache.org> wrote: > > > > > > > > > > CAUTION: This email originated from outside of the > > organization. Do > > > > > not click links or open attachments unless you can confirm > > the sender and > > > > > know the content is safe. > > > > > > > > > > > > > > > > > > > > +1 thanks for driving this FLIP. > > > > > > > > > > We actually have an internally forked equivalent of > > this, which is on > > > > > our > > > > > list to try to upstream. Your proposal would *almost* > > work "off the > > > > > shelf" > > > > > for us. We have an inverted order or priority: > > > > > - Rest API / Flink CLI > User Code > Cluster Config > > > > > > > > > > This is because our end users do not submit their jobs > > directly, our > > > > > service does it on their behalf. For our usecase we do > > not want to > > > > > allow > > > > > users to override certain values we set, since some are > > managed from > > > > > our > > > > > service configuration, example: checkpointing interval. > > > > > > > > > > How would you feel about including a cluster > > configuration to invert > > > > > the > > > > > order? This could be decoupled to a follow-up FLIP. > > > > > > > > > > Thanks, > > > > > Danny Cranmer > > > > > > > > > > > > > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen < > > jam.gz...@gmail.com> > > > > > wrote: > > > > > > > > > > > This is a good suggestion, we can start this work > > after we finish the > > > > > > discussion and vote > > > > > > > > > > > > -- > > > > > > Best > > > > > > > > > > > > ConradJam > > > > > > > > > > > > > > > > > > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道： > > > > > > > > > > > > > I've created a new ticket [FLINK-28973] Extending > > > > > /jars/:jarid/plan API > > > > > > to > > > > > > > support setting Flink configs - ASF JIRA ( > apache.org > > )< > > > > > > > https://issues.apache.org/jira/browse/FLINK-28973 > >. > > for the job > > > > > plan > > > > > > API. > > > > > > > Maybe we can create a new umbrella ticket for > > FLIP-256 and put > > > > > > FLINK-27060 > > > > > > > & FLINK-28973 as subtasks. WDYT? > > > > > > > > > > > > > > Best, > > > > > > > Zhanghao Chen > > > > > > > ________________________________ > > > > > > > From: zhengyu chen <jam.gz...@gmail.com> > > > > > > > Sent: Monday, August 15, 2022 18:06 > > > > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic > > Parameter With > > > > > Flink Rest > > > > > > > Api > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > We would like to start a discussion thread on > > FLIP-256 Support Job > > > > > > Dynamic > > > > > > > Parameter With Flink Rest Api > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > After the user submits the jar package, running a > > job through > > > > > restapi > > > > > > > (/jars/:jarid/run) [2] can only pass in > > (allowNonRestoredState, > > > > > > > savepointPath, programArg, entry-class, > parallelism) > > parameters, > > > > > which is > > > > > > > obvious with the diversification of job parameters > > (eg Checkpoint > > > > > > address) > > > > > > > > > > > > > > This solves the problem that the user can pass in > > other parameters > > > > > when > > > > > > > submitting a job, avoiding the user to define these > > job parameters > > > > > in the > > > > > > > code, resulting in the need to repackage the job > for > > each > > > > > modification > > > > > > > > > > > > > > There was some interest from users [3] from a > meetup > > and the > > > > > mailing > > > > > > list. > > > > > > > Looking forward to comments and feedback, thanks! > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run > > > > > > > > > > > > > > [3] > > https://issues.apache.org/jira/browse/FLINK-27060 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Best > > > > > > > > > > > > > > ConradJam > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Best > > > > ConradJam > > > > > > -- > Best > > ConradJam >
