Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Everyone, we have 2 bounding votes and 2 unbounding votes in our voting (url: https://lists.apache.org/thread/89f7kwnqgs8mfwos0h6b27lt3xn7383t). According to the rules, we need at least 3 bound votes to start this FLIP. If you forget to vote, it doesn't matter, we still have time to vote it, let's

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 On Sun, Sep 11, 2022 at 8:04 PM Zheng Yu Chen wrote: > Now FLIP-256 is start vote in this thread url: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread_89f7kwnqgs8mfwos0h6b27lt3xn7383t&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Now FLIP-256 is start vote in this thread url: https://lists.apache.org/thread/89f7kwnqgs8mfwos0h6b27lt3xn7383t zhengyu chen 于2022年8月15日周一 18:06写道： > > Hi all, > > > We would like to start a discussion thread on FLIP-256 Support Job > Dynamic Parameter With Flink Rest Api >

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

That sounds good to me. Thanks for working in this Zheng Yu! Hong From: Zheng Yu Chen Sent: 01 September 2022 05:08:03 To: dev@flink.apache.org Subject: RE: [EXTERNAL][DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api CAUTION: This emai

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Hi @Hong As @Chesnay said, we just follow the same order that the CLI currently enforces. @ Gyula and me agree this sugget， It also makes sense to mention that this FLIP is not about the prioritization of config options. so this feature maybe do this filp and after discuss , not now After the a

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

 Ok, I think that’s fair - overriding user config expands the scope of the FLIP, should cover CLI and REST, and can be split into another discussion. Since the motivation is to make the REST API have parity with the CLI, happy to defer further work to a different thread. @Chesnay, to clarif

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 for Chesnay's suggestion and taking a simple pragmatic approach. Gyula On Mon, Aug 29, 2022 at 3:41 PM Chesnay Schepler wrote: > TBH I don't really get why we're discussion the prioritization of config > options vs. what users have set in their application in this thread. > > Why don't we ju

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

TBH I don't really get why we're discussion the prioritization of config options vs. what users have set in their application in this thread. Why don't we just follow the same order that the CLI currently enforces? Same point regarding the external vs internal parameters. Ultimately this FLIP

RE: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

> +1, this will make external systems connected to Flink more efficient. > Thanks for driving the feature. Best, Manson On 2022/08/15 10:06:18 zhengyu chen wrote: > Hi all, > > > We would like to start a discussion thread on FLIP-256 Support Job Dynamic > Parameter With Flink Re

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Hi Hong: I compared @gyula and yours solutions, here are some of my thoughts: We can't control whether the user defines some parameters in the code, so I think User Code is always bigger than other configuration items (usually code in development has the highest priority) such as @gyula said, I t

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Originally I assumed that user code programmatic config changes always have highest priority in Flink. If I understand correctly this is true in other deployment modes . So I wouldn’t change that with a flag, the question is whether cluster config or rest api config has a higher priority. I think

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Hi Hong: Well, I think this should be the same idea as Gyula said before @Gyula What do you think ? If it is ok I will add this part to the documentation,Then the next process should be initiated by the community to vote this filp, I don't know who will host this part? Can someone help me with th

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Hi Zheng Yu, Sorry for the late reply, I was on holiday last week. Could we propose the following config instead? rest.submit.job.override-config = true/false - true: REST API will have priority over user code (i.e. Rest API / Flink CLI > User Code > Cluster Config) - false (default): user

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

HI Hong, Maybe you forgot. I don’t know if the current FLIP still needs to be modified. If not, we will start with the current plan. If you have relevant ideas in the future, please continue to discuss. If not, we will open voting at a later date. Teoh, Hong 于2022年8月22日周一 16:02写道： > > Hi Zheng Y

RE: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1, Lgtm, this proposal may helps me to manage the job more conveniently On 2022/08/15 10:06:18 zhengyu chen wrote: > Hi all, > > > We would like to start a discussion thread on FLIP-256 Support Job Dynamic > Parameter With Flink Rest Api >

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Hi Hong, If we want to distinguish between "cluster configuration" and "job configuration", I think the document will become complicated. What if we can ensure that this part of the configuration does not take effect or is intercepted? When the user deploys Session Cluster, if the checkpoint inter

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

Hi Zheng Yu, We would have to take the same "cluster configuration" (cannot be set on job submission) vs "job configuration" approach in the User code as well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config as a cluster configuration. That way, neither the REST API / User

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

@Gyula say that add a config for this but not expose it on the rest api is right when user start up session cluster,we can config (eg jobmanager.rest.api.submit.job.allow-reset-config = true/false) to controller this behavior Now we have an existing cluster, the admin configures checkpointing in

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

I don't think this is a good idea because it breaks the jar run api, this is just a job submit parameter, now a non-job parameter (overwriteFlinkConfiguration) is introduced. flinkConfiguration is just an extra extension, users can keep their submit job by original way Jing Ge 于2022年8月19日周五 07:0

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 for the FLIP. I like this FLIP. It is very useful. I'd like to ask one more question: " - The internal parameters passed in by the user contains the external parameter. At this point, we can see that the user configures the same two parameters in the *external parameter* and *interna

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

@Gyula: I was thinking of the use case where the managed platform also handles the job submission. Setup #1 External user : Creating jar Managed platform : Submit job Managed platform : Manage Flink cluster As compared to Setup #2 External user : Creating

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 for the proposal. @Hong : I feel that the inverted override flag should be a cluster setting and not something the user can override at will. I fear that this might defeat the purpose of the feature itself. So I think we should add a config for this but not expose it on the rest api Gyula On

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 to this FLIP. This is very useful for teams building a Flink platform to run jobs from an external user +1 on Danny's comment on adding a configuration to allow inverted order of overrides. However, might it be better to include an "override" toggle in the REST API itself? That way we can

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 thanks for driving this FLIP. We actually have an internally forked equivalent of this, which is on our list to try to upstream. Your proposal would *almost* work "off the shelf" for us. We have an inverted order or priority: - Rest API / Flink CLI > User Code > Cluster Config This is because

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

This is a good suggestion, we can start this work after we finish the discussion and vote -- Best ConradJam 于2022年8月15日周一 21:01写道： > I've created a new ticket [FLINK-28973] Extending /jars/:jarid/plan API to > support setting Flink configs - ASF JIRA (apache.org)< > https://issues.apache.org/

Re:[DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 Thanks for driving this,It makes the REST API that runs the job more powerful and complete Best JasonLee Replied Message | From | zhengyu chen | | Date | 08/15/2022 18:06 | | To | | | Subject | [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api | Hi all, We w

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

I've created a new ticket [FLINK-28973] Extending /jars/:jarid/plan API to support setting Flink configs - ASF JIRA (apache.org). for the job plan API. Maybe we can create a new umbrella ticket for FLIP-256 and put FLINK-27060 & FLINK-28973 as

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1. Thanks for the effort. I think we should also extend the jarid-plan API [1] in a similar way, as some configurations might affect the jog graph as well. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-plan Best, Zhanghao Chen _

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1, I think this is a very useful proposal, especially for session mode. And for simplicity, maybe we can treat the job-level configurations and cluster-level configurations identically, as your mentioned, the cluster-level configurations will be ignored. Best, Yanfei zhengyu chen 于2022年8月15日周一 1