+1 for Chesnay's suggestion and taking a simple pragmatic approach. Gyula
On Mon, Aug 29, 2022 at 3:41 PM Chesnay Schepler <ches...@apache.org> wrote: > TBH I don't really get why we're discussion the prioritization of config > options vs. what users have set in their application in this thread. > > Why don't we just follow the same order that the CLI currently enforces? > Same point regarding the external vs internal parameters. > > Ultimately this FLIP is only about getting closer to achieving feature > parity between the CLI and jar submission, so let's focus on that. > Everything else should cover both the REST API and CLI, and would thus > be out-of-scope of this FLIP imo. > > On 29/08/2022 03:52, Zheng Yu Chen wrote: > > Hi Hong: > > > > I compared @gyula and yours solutions, here are some of my thoughts: > > > > We can't control whether the user defines some parameters in the code, > so I > > think User Code is always bigger than other configuration items (usually > > code in development has the highest priority) > > such as @gyula said, I think it's more reasonable to put User code > > first. Then, according to the cluster configuration > > (rest.submit.job.override-config = true/false), whether this job > parameter > > is configured to determine which value should take effect > > > > If we follow your strategy（Rest API / Flink CLI > User Code > Cluster > > Config）, it means that after we submit the job, the user's hard-coded > > configuration will be invalid, which will make the user feel strange: I > > have configured this parameter in the code, why does it not take effect? > > So I will be more inclined to support @gyula suggestion > > > > (Default) override true: > > User code > rest api > cluster conf > > > > Override false : > > User code > cluster conf > rest api > > > > @Hong What do you think ? > > > > > > > > Teoh, Hong <lian...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道： > > > >> Hi Zheng Yu, > >> > >> Sorry for the late reply, I was on holiday last week. > >> > >> Could we propose the following config instead? > >> > >> rest.submit.job.override-config = true/false > >> - true: REST API will have priority over user code (i.e. Rest API / > >> Flink CLI > User Code > Cluster Config) > >> - false (default): user code will have priority over REST API (i.e. > User > >> Code > Rest API / Flink CLI > Cluster Config) > >> > >> This way, the default behaviour will be according to the proposed FLIP, > >> but we will have an additional toggle to ignore the configuration set in > >> user code. > >> > >> > However, there will be a problem in doing this. If the user > writes > >> > this Flink Config in jar by hardcoding, the highest priority > >> must be > >> > the code at this time, so the problem may still exist, and > the > >> user > >> > cannot be prevented from this behavior > >> > >> Hmm, would it be better if we set it such that the > >> "rest.submit.job.override-config" cannot be overwritten in user code > >> (ignored), just like configuration "jobmanager.rpc.port"? > >> > >> What do you think? > >> > >> Regards, > >> Hong > >> > >> > >> > >> On 26/08/2022, 12:05, "Zheng Yu Chen" <jam.gz...@gmail.com> wrote: > >> > >> CAUTION: This email originated from outside of the organization. Do > >> not click links or open attachments unless you can confirm the sender > and > >> know the content is safe. > >> > >> > >> > >> HI Hong, > >> > >> Maybe you forgot. I don’t know if the current FLIP still needs to > be > >> modified. If not, we will start with the current plan. If you have > >> relevant ideas in the future, please continue to discuss. If not, > we > >> will open voting at a later date. > >> > >> Teoh, Hong <lian...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道： > >> > > >> > Hi Zheng Yu, > >> > > >> > We would have to take the same "cluster configuration" (cannot be > >> set on job submission) vs "job configuration" approach in the User code > as > >> well. And we can classify > jobmanager.rest.api.submit.job.allow-reset-config > >> as a cluster configuration. That way, neither the REST API / User code > can > >> override this configuration, and it can only be set in cluster > >> configuration on startup. > >> > > >> > There are other cluster specific configurations that are already > >> treated this way (e.g. jobmanager.rpc.address, > >> jobmanager.memory.process.size). As part of this work, I wonder if we > can > >> update the Flink docs on configuration ( > >> > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ > ) > >> to specify which configuration is "cluster" and which is "job". > >> > > >> > Regards, > >> > Hong > >> > > >> > > >> > > >> > On 20/08/2022, 09:16, "Zheng Yu Chen" <jam.gz...@gmail.com> > wrote: > >> > > >> > CAUTION: This email originated from outside of the > organization. > >> Do not click links or open attachments unless you can confirm the sender > >> and know the content is safe. > >> > > >> > > >> > > >> > @Gyula say that add a config for this but not expose it on > the > >> rest > >> > api is right > >> > when user start up session cluster,we can config (eg > >> > jobmanager.rest.api.submit.job.allow-reset-config = > true/false) > >> to > >> > controller this behavior > >> > > >> > Now we have an existing cluster, the admin configures > >> checkpointing > >> > interval by this session cluster > >> > > >> > jobmanager.rest.api.submit.job.allow-reset-config = > true/false > >> > > >> > - true : allow user reset new value config > >> > - false(default) : not allow user set new config from ,throw > exp > >> to > >> > tell user what properties is admin was set it > >> > > >> > However, there will be a problem in doing this. If the user > >> writes > >> > this Flink Config in jar by hardcoding, the highest priority > >> must be > >> > the code at this time, so the problem may still exist, and > the > >> user > >> > cannot be prevented from this behavior. > >> > > >> > what do you think ? > >> > > >> > @Hong @Gyula @Teoh @Danny > >> > > >> > > >> > Gyula Fóra <gyula.f...@gmail.com> 于2022年8月18日周四 18:37写道： > >> > > > >> > > +1 for the proposal. > >> > > > >> > > @Hong : I feel that the inverted override flag should be a > >> cluster setting > >> > > and not something the user can override at will. I fear > that > >> this might > >> > > defeat the purpose of the feature itself. > >> > > So I think we should add a config for this but not expose > it > >> on the rest api > >> > > > >> > > Gyula > >> > > > >> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong > >> <lian...@amazon.co.uk.invalid> > >> > > wrote: > >> > > > >> > > > +1 to this FLIP. > >> > > > This is very useful for teams building a Flink platform > to > >> run jobs from > >> > > > an external user > >> > > > > >> > > > +1 on Danny's comment on adding a configuration to allow > >> inverted order of > >> > > > overrides. > >> > > > However, might it be better to include an "override" > toggle > >> in the REST > >> > > > API itself? That way we can change the flink > configuration > >> override > >> > > > behaviour without restarting the Flink cluster. This > would > >> make sense if we > >> > > > are thinking of a Session cluster and deploying multiple > >> Flink jobs to the > >> > > > same cluster. > >> > > > > >> > > > Regards, > >> > > > Hong > >> > > > > >> > > > > >> > > > On 18/08/2022, 10:46, "Danny Cranmer" < > >> dannycran...@apache.org> wrote: > >> > > > > >> > > > CAUTION: This email originated from outside of the > >> organization. Do > >> > > > not click links or open attachments unless you can > confirm > >> the sender and > >> > > > know the content is safe. > >> > > > > >> > > > > >> > > > > >> > > > +1 thanks for driving this FLIP. > >> > > > > >> > > > We actually have an internally forked equivalent of > >> this, which is on > >> > > > our > >> > > > list to try to upstream. Your proposal would *almost* > >> work "off the > >> > > > shelf" > >> > > > for us. We have an inverted order or priority: > >> > > > - Rest API / Flink CLI > User Code > Cluster Config > >> > > > > >> > > > This is because our end users do not submit their > jobs > >> directly, our > >> > > > service does it on their behalf. For our usecase we > do > >> not want to > >> > > > allow > >> > > > users to override certain values we set, since some > are > >> managed from > >> > > > our > >> > > > service configuration, example: checkpointing > interval. > >> > > > > >> > > > How would you feel about including a cluster > >> configuration to invert > >> > > > the > >> > > > order? This could be decoupled to a follow-up FLIP. > >> > > > > >> > > > Thanks, > >> > > > Danny Cranmer > >> > > > > >> > > > > >> > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen < > >> jam.gz...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > This is a good suggestion, we can start this work > >> after we finish the > >> > > > > discussion and vote > >> > > > > > >> > > > > -- > >> > > > > Best > >> > > > > > >> > > > > ConradJam > >> > > > > > >> > > > > > >> > > > > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道： > >> > > > > > >> > > > > > I've created a new ticket [FLINK-28973] Extending > >> > > > /jars/:jarid/plan API > >> > > > > to > >> > > > > > support setting Flink configs - ASF JIRA ( > apache.org > >> )< > >> > > > > > > https://issues.apache.org/jira/browse/FLINK-28973>. > >> for the job > >> > > > plan > >> > > > > API. > >> > > > > > Maybe we can create a new umbrella ticket for > >> FLIP-256 and put > >> > > > > FLINK-27060 > >> > > > > > & FLINK-28973 as subtasks. WDYT? > >> > > > > > > >> > > > > > Best, > >> > > > > > Zhanghao Chen > >> > > > > > ________________________________ > >> > > > > > From: zhengyu chen <jam.gz...@gmail.com> > >> > > > > > Sent: Monday, August 15, 2022 18:06 > >> > > > > > To: dev@flink.apache.org <dev@flink.apache.org> > >> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic > >> Parameter With > >> > > > Flink Rest > >> > > > > > Api > >> > > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > > >> > > > > > We would like to start a discussion thread on > >> FLIP-256 Support Job > >> > > > > Dynamic > >> > > > > > Parameter With Flink Rest Api > >> > > > > > < > >> > > > > > > >> > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > >> > > > > > > > >> > > > > > [1] > >> > > > > > > >> > > > > > After the user submits the jar package, running a > >> job through > >> > > > restapi > >> > > > > > (/jars/:jarid/run) [2] can only pass in > >> (allowNonRestoredState, > >> > > > > > savepointPath, programArg, entry-class, > parallelism) > >> parameters, > >> > > > which is > >> > > > > > obvious with the diversification of job > parameters > >> (eg Checkpoint > >> > > > > address) > >> > > > > > > >> > > > > > This solves the problem that the user can pass in > >> other parameters > >> > > > when > >> > > > > > submitting a job, avoiding the user to define > these > >> job parameters > >> > > > in the > >> > > > > > code, resulting in the need to repackage the job > for > >> each > >> > > > modification > >> > > > > > > >> > > > > > There was some interest from users [3] from a > meetup > >> and the > >> > > > mailing > >> > > > > list. > >> > > > > > Looking forward to comments and feedback, thanks! > >> > > > > > > >> > > > > > > >> > > > > > [1] > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > >> > > > > > > >> > > > > > > >> > > > > > [2] > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run > >> > > > > > > >> > > > > > [3] > >> https://issues.apache.org/jira/browse/FLINK-27060 > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > Best > >> > > > > > > >> > > > > > ConradJam > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > >> > >> -- > >> Best > >> > >> ConradJam > >> > >> > >
