Hi Guys:
FYI, I have integrated your comments and drawn the FLIP-215[1], I will
create another thread to vote for it.
[1]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-215%3A+Introduce+FlinkSessionJob+CRD+in+the+kubernetes+operator
Best,
Aitozi.
Aitozi <gjying1...@gmail.com> 于2022年3月17日周四 11:16写道：
> Hi Biao Geng:
>
> Thanks for your feedback, I'm +1 to go with option#2. It's a good
> point that
>
> we should improve the error message debugging for the session job, I
> think
>
> it can be a follow up work as an improvement after we support the session
> job operation.
>
>
>
> Best,
>
> Aitozi.
>
>
> Geng Biao <biaoge...@gmail.com> 于2022年3月17日周四 10:55写道：
>
>> Thanks Aitozi for the work!
>>
>> I lean to option#2 of using JarRunHeaders with uber job jar as well. As
>> Yang said, the user defined dependencies may be better supported in
>> upstream flink.
>> A follow-up thought: I think we should care the potential influence on
>> user experiences: as the job graph is generated in JM, when the generation
>> fails due to some issues in the main() method, we should do some work on
>> showing such error messages in this proposal or the later k8s operator
>> implementation. Reason for this question is that if users submit many jobs
>> to one same session cluster, it may be not easy for them to find relevant
>> error logs about main() method of a specific job. The FLINK-25715 could
>> help us later.
>>
>>
>> Best,
>> Biao Geng
>>
>>
>> 发件人: Aitozi <gjying1...@gmail.com>
>> 日期: 星期三, 2022年3月16日 下午5:19
>> 收件人: dev@flink.apache.org <dev@flink.apache.org>
>> 主题: Re: [DISCUSS] Support the session job management in kubernetes
>> operator
>> Hi Yang Wang
>> Thanks for your feedback, Provide the local and http implementation
>> for
>> the first version makes sense to me.
>> +1 for it.
>>
>> Best,
>> Aitozi
>>
>> Yang Wang <danrtsey...@gmail.com> 于2022年3月16日周三 16:44写道：
>>
>> > # How to download the user jars
>> > I agree with Gyula that it will be a burden if we bundle the flink
>> > filesystem dependencies in the operator image.
>> > Maybe we could have a *ArtifactFetcher* interface in the
>> > flink-kubernetes-operator. By default, we provide the local and http
>> > implementation,
>> > which means we could get the user jars from local files or HTTP URLs.
>> Flink
>> > filesystem support could be done as a follow-up based on the feedback.
>> >
>> > If the user wants to use the local implementation, they need to mount a
>> > PV(aka persist volume) to the operator first and then put their jars
>> into
>> > the PV.
>> >
>> > # How to talk to session JobManager to submit the job
>> > After more consideration, I also prefer the second approach, via REST
>> API
>> > /jars/:jarid/run. If we have strong requirements to support dependencies
>> > jars and
>> > artifacts, we could try to support this in the upstream project.
>> >
>> > Best,
>> > Yang
>> >
>> >
>> > Aitozi <gjying1...@gmail.com> 于2022年3月16日周三 16:11写道：
>> >
>> > > Hi Gyula
>> > > Thanks for your quick response. Regarding the different
>> filesystems
>> > > dependency,
>> > > I think we can make it optional and pluggable, and let it choose by
>> user
>> > > when building
>> > > their operator image. Users can build their image from the base
>> operator
>> > > image and
>> > > add filesystem dependency they want to use to it. BTW, we can support
>> the
>> > > http URI
>> > > by default.
>> > >
>> > > Thanks,
>> > > Aitozi.
>> > >
>> > > Gyula Fóra <gyula.f...@gmail.com> 于2022年3月16日周三 15:53写道：
>> > >
>> > > > Thank you Aitozi!
>> > > >
>> > > > I think this will be a very nice (and simple) addition to enable
>> these
>> > > > use-cases.
>> > > >
>> > > > I have 2 comments regarding the proposal:
>> > > >
>> > > > 1. I think if we want to support different filesystems to download
>> jars
>> > > > from, we probably need some clever ways to add external operator
>> > > > dependencies (jars, configs).
>> > > > I would prefer not to bundle them into the base operator image.
>> > > >
>> > > > 2. I think we should avoid creating the jobgraphs on the operator
>> side
>> > > and
>> > > > use the jar upload/run rest api instead as you suggested. This will
>> > avoid
>> > > > flink version and dependency conflicts.
>> > > >
>> > > > Cheers,
>> > > > Gyula
>> > > >
>> > > > On Wed, Mar 16, 2022 at 8:41 AM Aitozi <gjying1...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi Guys:
>> > > > >
>> > > > > I would like to open a discussion for support session job
>> > > management
>> > > > in
>> > > > > kubernetes operator. It’s intended to enhance the
>> > > > flink-kubernetes-operator
>> > > > > to manage the session job with k8s tooling. I have drafted the
>> design
>> > > > > doc[1]. Please refer to it and give me some feedback .
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1WPGbur1eT3H_5gN-kyXfp7EDjdbJUURx6jN8nt6UT-s/edit#
>> <
>> https://docs.google.com/document/d/1WPGbur1eT3H_5gN-kyXfp7EDjdbJUURx6jN8nt6UT-s/edit
>> >
>> > > > >
>> > > > > Best,
>> > > > >
>> > > > > Aitozi.
>> > > > >
>> > > >
>> > >
>> >
>>
>
