I also lean to persist the FlinkDeployment and its status via K8s resources.
Unless necessary, we should not introduce other external dependencies(e.g.
mysql).
It will make the k8s operator more complicated.
Best,
Yang
Gyula Fóra 于2022年2月21日周一 02:48写道：
> Hi!
>
> Thank you for your interest in
Hi!
Thank you for your interest in contributing to the operator.
The operator persists information in the status of the FlinkDeployment
resource. We should not need any additional persistence layer on top of
this in the current design.
Could you please give me a concrete example of what is not w
Hi, regarding the construction of k8s Flink Operator, I have already
completed some functions. I hope to contribute this part of the functions
and discuss with the community how to improve it. How should I start?
So far I have seen that the component has no operation persistence. Should
we persist
Hi Konstantin,
I completely agree with the general philosophy that if a resource exists it
should be "running" or I would rather say "do it's thing" whatever that
means for a particular resource.
We followed this design principle when we decided to have only 2 "desired
states" running and suspend
Hi Gyula,
sorry for joining late. One comment on the API design for consideration: we
are using the job.state as kind of a "desired state", right? This is quite
uncommon in Kubernetes to my knowledge. In Kubernetes almost always the
fact that a resource exists means that it should be "running". Th
Hi All!
Thank you all for reviewing the PR and already helping to make it better. I
have opened a bunch of jira tickets under
https://issues.apache.org/jira/browse/FLINK-25963 based on some comments
and incomplete features in general.
Given that there were no major objections about the prototype,
Thanks for the explanation.
Given that it is unrelated with java version in Flink.
Starting with java11 for the flink-kubernetes-operator makes sense to me.
Best,
Yang
Thomas Weise 于2022年2月15日周二 23:57写道：
> Hi,
>
> At this point I see no reason to support Java 8 for a new project.
> Java 8 is b
Hi Gyula!
Alright! For clarity, the savepoint path and other savepoint related
configuration can be put into flinkConfiguration.
On the side, I think the automatic savepoint generation for instance should
be put into JobSpec along with other job options, and FlinkConfiguration
only configures what
Hi,
At this point I see no reason to support Java 8 for a new project.
Java 8 is being phased out, we should start with 11.
Also, since the operator isn't a library but effectively just a docker
image, the ability to change the Java version isn't as critical as it
is for Flink core, which needs t
Good catch, Yang Wang and Gyula on the Java version. I personally prefer
that we simply can not support Java 8 for the operator, since it is a net
new project we are better off starting support at Java 11 right away.
As Gyula outlined above, it is important to note that it only affects the
operato
Hi Devs,
Yang Wang discovered that the current prototype is not compatible with Java
8 but only 11 and upwards.
The reason for this is that the java operator SDK itself is not java 8
compatible unfortunately.
Given that Java 8 is on the road to deprecation and that the operator runs
as a contain
I also lean to not introduce the savepoint/checkpoint related fields to the
job spec, especially in the very beginning of flink-kubernetes-operator.
Best,
Yang
Gyula Fóra 于2022年2月15日周二 19:02写道：
> Hi Peng Yuan!
>
> While I do agree that savepoint path is a very important production
> configurat
Hi Peng Yuan!
While I do agree that savepoint path is a very important production
configuration there are a lot of other things that come to my mind:
- savepoint dir
- checkpoint dir
- checkpoint interval/timeout
- high availability settings (provider/storagedir etc)
just to name a few...
Wh
Hi Matyas!
Thanks for your reply!
For 1. and 3. scenarios，I couldn't agree more with the podTemplate solution
, i missed this part.
For savepoint related configuration, I think it's very important to be
specified in JobSpec, Because savepoint is a very common configuration for
upgrading a job, if
Hi Peng,
Thanks for your feedback. Regarding 1. and 3. scenarios, the podTemplate
functionality in the operator could cover both. We also need to be careful
about introducing proxy parameters in the CRD spec. The savepoint path is
usually accompanied with a bunch of other configurations for exampl
Hi Gyula!
I have reviewed the prototype design of flink-kubernetes-operator you
submitted, and I have the following questions:
1.Can a Flink Jar package that supports pulling from the sidecar be added
to the JobSpec? just like this:
> initContainers:
> - name: downloader
> image: c
Hi Flink Devs!
We would like to present to you the first prototype of the
flink-kubernetes-operator that was built based on the FLIP and the
discussion on this mail thread. We would also like to call out some design
decisions that we have made regarding architecture components that were not
explic
Hi Gyula,
Thanks!
It's great to see the project getting started and I can't wait to see the
PR and start contributing code.😄😄😄
Best Wishes!
Peng Yuan
On Mon, Feb 14, 2022 at 4:14 PM Gyula Fóra wrote:
> Hi Peng Yuan!
>
> The repo is already created:
> https://github.com/apache/flink-kubernetes-
Hi Peng Yuan!
The repo is already created:
https://github.com/apache/flink-kubernetes-operator
We will open the PR with the initial prototype later today, stay tuned in
this thread! :)
Cheers,
Gyula
On Mon, Feb 14, 2022 at 9:09 AM K Fred wrote:
> Hi All,
>
> Has the project of flink-kubernete
Hi All,
Has the project of flink-kubernetes-operator been created in github?
Peng Yuan
On Wed, Feb 9, 2022 at 1:23 AM Gyula Fóra wrote:
> I agree with flink-kubernetes-operator as the repo name :)
> Don't have any better idea
>
> Gyula
>
> On Sat, Feb 5, 2022 at 2:41 AM Thomas Weise wrote:
>
I agree with flink-kubernetes-operator as the repo name :)
Don't have any better idea
Gyula
On Sat, Feb 5, 2022 at 2:41 AM Thomas Weise wrote:
> Hi,
>
> Thanks for the continued feedback and discussion. Looks like we are
> ready to start a VOTE, I will initiate it shortly.
>
> In parallel it wo
he discussion in [2] might be
> > relevant.
> >
> > [1]
> > https://github.com/MicroFocus/opsb-flink-k8s-operator/blob/main/kubernetes/sample_batch_job.yaml
> > [2] https://issues.apache.org/jira/browse/FLINK-24150
> >
> > Regards,
> > Alexis.
> >
> &g
thub.com/MicroFocus/opsb-flink-k8s-operator/blob/main/kubernetes/sample_batch_job.yaml
> [2] https://issues.apache.org/jira/browse/FLINK-24150
>
> Regards,
> Alexis.
>
> -Original Message-
> From: K Fred
> Sent: Montag, 7. Februar 2022 09:36
> To: dev@flink.apa
ain/kubernetes/sample_batch_job.yaml
[2] https://issues.apache.org/jira/browse/FLINK-24150
Regards,
Alexis.
-Original Message-
From: K Fred
Sent: Montag, 7. Februar 2022 09:36
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator
Hi Gyula!
You are right. I thi
gt; On Sat, Feb 5, 2022 at 12:28 PM Alexis Sarda-Espinosa <
> > > > alexis.sarda-espin...@microfocus.com> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > just wanted to mention that my employer agreed to open source the
> > just wanted to mention that my employer agreed to open source the
> PoC I
> > > > developed: https://github.com/MicroFocus/opsb-flink-k8s-operator
> > > >
> > > > I understand the concern for maintainability, so Gradle & Kotlin
> might
> >
nability, so Gradle & Kotlin might
> > not
> > > be appealing to you, but at least it gives you another reference. The
> > Helm
> > > resources in particular might be useful.
> > >
> > > There are bits and pieces there referring to Flink sessions, but those
s and pieces there referring to Flink sessions, but those
> are
> > just placeholders, the functioning parts use application mode with native
> > integration.
> >
> > Regards,
> > Alexis.
> >
> >
> > From: Thomas
ust placeholders, the functioning parts use application mode with native
> integration.
>
> Regards,
> Alexis.
>
>
> From: Thomas Weise
> Sent: Saturday, February 5, 2022 2:41 AM
> To: dev
> Subject: Re: [DISCUSS] FLIP-212: Introduce Fl
rday, February 5, 2022 2:41 AM
To: dev
Subject: Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator
Hi,
Thanks for the continued feedback and discussion. Looks like we are
ready to start a VOTE, I will initiate it shortly.
In parallel it would be good to find the repository name.
Hi,
Thanks for the continued feedback and discussion. Looks like we are
ready to start a VOTE, I will initiate it shortly.
In parallel it would be good to find the repository name.
My suggestion would be: flink-kubernetes-operator
I thought "flink-operator" could be a bit misleading since the t
Hi Danny,
So far we have been focusing our dev efforts on the initial native
implementation with the team.
If the discussion and vote goes well for this FLIP we are looking forward
to contributing the initial version sometime next week (fingers crossed).
At that point I think we can already start
I have been discussing this one with my team. We are interested in the
Standalone mode, and are willing to contribute towards the implementation.
Potentially we can work together to support both modes in parallel?
Thanks,
On Wed, Feb 2, 2022 at 4:02 PM Gyula Fóra wrote:
> Hi Danny!
>
> Thanks f
Hi Danny!
Thanks for the feedback :)
Versioning:
Versioning will be independent from Flink and the operator will depend on a
fixed flink version (in every given operator version).
This should be the exact same setup as with Stateful Functions (
https://github.com/apache/flink-statefun). So indepe
Hey team,
Great work on the FLIP, I am looking forward to this one. I agree that we
can move forward to the voting stage.
I have general feedback around how we will handle job submission failure
and retry. As discussed in the Rejected Alternatives section, we can use
Java to handle job submission
Hi team,
Thank you for the great feedback, Thomas has updated the FLIP page
accordingly. If you are comfortable with the currently existing design and
depth in the FLIP [1] I suggest moving forward to the voting stage - once
that reaches a positive conclusion it lets us create the separate code
re
HI Xintong,
Thanks for the feedback and please see responses below -->
On Fri, Jan 28, 2022 at 12:21 AM Xintong Song wrote:
> Thanks Thomas for drafting this FLIP, and everyone for the discussion.
>
> I also have a few questions and comments.
>
> ## Job Submission
> Deploying a Flink session cl
Thanks Thomas for drafting this FLIP, and everyone for the discussion.
I also have a few questions and comments.
## Job Submission
Deploying a Flink session cluster via kubectl & CR and then submitting jobs
to the cluster via Flink cli / REST is probably the approach that requires
the least effor
Thanks for the feedback!
>
> # 1 Flink Native vs Standalone integration
> Maybe we should make this more clear in the FLIP but we agreed to do the
> first version of the operator based on the native integration.
> While this clearly does not cover all use-cases and requirements, it seems
> this wo
Hi All!
Thanks for the questions, there are still quite a few unknowns and
decisions to be made but here are my current thoughts:
# 1 Flink Native vs Standalone integration
Maybe we should make this more clear in the FLIP but we agreed to do the
first version of the operator based on the native i
Hi Biao,
# 1 Flink Native vs Standalone integration
I think we have got a trend in this discussion[1] that the newly introduced
Flink K8s operator will start with native K8s integration first.
Do you have some concerns about this?
# 2 K8S StatefulSet v.s. K8S Deployment
IIUC, the FlinkDeployment
Hi Thomas,
Thanks a lot for the great efforts in this well-organized FLIP! After
reading the FLIP carefully, I think Yang has given some great feedback and
I just want to share some of my concerns:
# 1 Flink Native vs Standalone integration
I believe it is reasonable to support both modes in the lo
Thanks Thomas for creating FLIP-212 to introduce the Flink Kubernetes
Operator.
The proposal looks already very good to me and has integrated all the input
in the previous discussion(e.g. native K8s VS standalone, Go VS java).
I read the FLIP carefully and have some questions that need to be clar
Hi,
As promised in [1] we would like to start the discussion on the
addition of a Kubernetes operator to the Flink project as FLIP-212:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-212%3A+Introduce+Flink+Kubernetes+Operator
Please note that the FLIP is currently focussed on the overall
44 matches
Mail list logo
