Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Matyas! So to clarify your suggestion, we would have the following JobStatus fields: jobId : String state : String savepointInfo : SavepointInfo jobDetailsInfo : String (optional) - output of Flink Rest API job details And the user could configure with a flag whether to include jobDetailsInfo

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Gyula, since the jobDetailsInfo could evolve, another option would be to dump it as yaml/json into the metadata. Best, Matyas On Fri, Jul 15, 2022 at 2:58 PM Gyula Fóra wrote: > Based on some further though, a reasonable middleground would be to add an > optional metadata/jobDetailsInfo fie

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Based on some further though, a reasonable middleground would be to add an optional metadata/jobDetailsInfo field to the JobStatus. We would also add an accompanying config option (default false) whether to populate this field for jobs. This way operator users could decide if they want to expose t

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi All! I fully acknowledge the general need to access more info about the running deployments. This need however is very specific to the use-cases / platforms built on the operator. I think we need a good way to tackle this without growing the status arbitrarily. Currently the JobStatus in the o

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Martin, Yes, that's understandable. I think adding job endTime, duration, jobPlan is useful to other Flink users too as they now have info to track: 1. endTime: If the job has ended, the user can know when it has ended. If the job is still streaming, then the user can know as it defaults to

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Daren, Could you list the benefits for the users of Flink? I do think that an internal AWS requirement is not a good argument for getting something done in Flink. Best regards, Martijn Op di 12 jul. 2022 om 21:17 schreef WONG, DAREN : > Hi Yang, > > The requirement to add *plan* currently o

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Yang, The requirement to add *plan* currently originates from an internal AWS requirement as our service needs visibility of *plan*, but we think it could be beneficial as well to customers who uses *plan* too. Regards, Daren On 12/07/2022, 13:23, "Yang Wang" wrote: CAUTION: This

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Thanks for the explanation. Only having 1 API call in most cases makes sense to me. Could you please elaborate more about why do we need the *plan* in CR status? Best, Yang Gyula Fóra 于2022年7月12日周二 17:36写道： > Hi Devs! > > I discussed with Daren offline, and I agree with him that technically w

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Devs! I discussed with Daren offline, and I agree with him that technically we almost never need 2 API calls. I think it's fine to have a second API call once directly after application submission (technically even this can be eliminated by setting a fix job id always). +1 from me. Cheers, G

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Matyas, Thanks for the feedback, and yes I agree. An alternative approach would instead be: - 2 API calls only when jobID is not available (i.e when submitting a new application cluster, which is a one-off event). - 1 API call when jobID is already available by directly calling "/jobs/:jobid

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

I share mytyas's concern if we list the jobs first and then followed by some get-job-detail requests. It is a bit expensive and I do not see the benefit to store jobPlan in the CR status. Best, Yang Őrhidi Mátyás 于2022年7月11日周一 21:43写道： > Hi Daren, > > At the moment the Operator fetches the job

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Daren, At the moment the Operator fetches the job state via https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobs-overview which contains the 'end-time' and 'duration' fields already. I feel calling the https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_ap

[DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi everyone, I am Daren from AWS Kinesis Data Analytics (KDA) team. I had a quick chat with Gyula as I propose to include a few additional fields in the jobStatus CRD for Flink Kubernetes Operator such as: - endTime - duration - jobPlan Further details of each states can be found here