Hi all, Thanks a lot for clarifying Yikun! I have no more concerns.
Best regards, Martijn Op vr 22 jul. 2022 om 10:42 schreef bo zhaobo <bzhaojyathousa...@gmail.com>: > Hi All, > > Thanks for all feedbacks from you. All of them are helpful and valuable for > us. > > If there is no further comment towards FLIP-250 we introduced, we plan to > setup a VOTE thread next Monday. > > Thank you all !! > > BR > > Bo Zhao > > > bo zhaobo <bzhaojyathousa...@gmail.com> 于2022年7月15日周五 10:02写道： > > > Thanks all, @Yang Wang and @Yikun Jiang. > > > > Hi Martijn, > > > > We understand your concern. And do the above emails clear your doubts? > > > > " > > Thanks for the info! I think I see that you've already updated the FLIP > to > > reflect how customized schedulers are beneficial for both batch and > > streaming jobs. > > " > > > > >>> > > > > Yeah, that's true that the "Motivation" paragraph makes readers confused. > > So > > I updated the FLIP description. And thanks for your feedback and correct. > > > > " > > The reason why I'm not too happy that we would only create a reference > > implementation for Volcano is that we don't know if the generic support > for > > customized scheduler plugins will also work for others. We think it will, > > but since there would be no other implementation available, we are not > > sure. My concern is that when someone tries to add support for another > > scheduler, we notice that we actually made a mistake or should improve > the > > generic support. > > " > > > > >>> > > > > Yeah, I understand your concern. Via YiKun Jinag's description and > > experience sharing, > > does he make you know more? Or we need to figure out the common part of > > some popular > > K8S customized schedulers and refresh the doc? Waiting for your advice. > > ;-) > > > > Best regards, > > > > Bo Zhao > > > > Yikun Jiang <yikunk...@gmail.com> 于2022年7月14日周四 18:45写道： > > > >> > And maybe we also could ping Yikun Jiang who has done similar things > in > >> Spark. > >> > >> Thanks for @wangyang ping. Yes, I was involved in Spark's customized > >> scheduler support work and as the main completer. > >> > >> For customized scheduler support, I can share scheduler's requirement in > >> here: > >> > >> 1. Help scheduler to *specify* the scheduler name > >> > >> 2. Help scheduler to create the* scheduler related > label/annotation/CRD*, > >> such as > >> - Yunikorn needs labels/annotations > >> < > >> > https://yunikorn.apache.org/docs/user_guide/labels_and_annotations_in_yunikorn/ > >> > > >> (maybe task group CRD in future or not) > >> - Volcano needs annotations and CRD < > https://volcano.sh/en/docs/podgroup/ > >> > > >> - Kube-batch needs annotations/CRD > >> <https://github.com/kubernetes-sigs/kube-batch/tree/master/config/crds> > >> - Kueue needs annotation support > >> < > >> > https://github.com/kubernetes-sigs/kueue/blob/888cedb6e62c315e008916086308a893cd21dd66/config/samples/sample-job.yaml#L6 > >> > > >> and > >> cluster level CRD > >> > >> 3. Help the scheduler to create the scheduler meta/CRD at the* right > >> time*, > >> such as if users want to avoid pod max pending, we need to create the > >> scheduler required CRD before pod creation. > >> > >> For complex requirements, Spark uses featurestep to support (looks flink > >> decorators are very similar to it) > >> For simple requirements, they can just use configuration or Pod > Template. > >> [1] > >> > >> > https://spark.apache.org/docs/latest/running-on-kubernetes.html#customized-kubernetes-schedulers-for-spark-on-kubernetes > >> > >> From the FLIP, I can see the above requirements are covered. > >> > >> BTW, I think Flink decorators' existing and new added interface have > >> already covered all requirements of Kubernetes, so I personally think > the > >> K8s related scheduler requirement can also be well covered by it. > >> > >> Regards, > >> Yikun > >> > >> > >> On Thu, Jul 14, 2022 at 5:11 PM Yang Wang <danrtsey...@gmail.com> > wrote: > >> > >> > I think we could go over the customized scheduler plugin mechanism > again > >> > with YuniKorn to make sure that it is common enough. > >> > But the implementation could be deferred. > >> > > >> > And maybe we also could ping Yikun Jiang who has done similar things > in > >> > Spark. > >> > > >> > For the e2e tests, I admit that they could be improved. But I am not > >> sure > >> > whether we really need the java implementation instead. > >> > This is out of the scope of this FLIP and let's keep the discussion > >> > under FLINK-20392. > >> > > >> > > >> > Best, > >> > Yang > >> > > >> > Martijn Visser <martijnvis...@apache.org> 于2022年7月14日周四 15:28写道： > >> > > >> > > Hi Bo, > >> > > > >> > > Thanks for the info! I think I see that you've already updated the > >> FLIP > >> > to > >> > > reflect how customized schedulers are beneficial for both batch and > >> > > streaming jobs. > >> > > > >> > > The reason why I'm not too happy that we would only create a > reference > >> > > implementation for Volcano is that we don't know if the generic > >> support > >> > for > >> > > customized scheduler plugins will also work for others. We think it > >> will, > >> > > but since there would be no other implementation available, we are > not > >> > > sure. My concern is that when someone tries to add support for > another > >> > > scheduler, we notice that we actually made a mistake or should > improve > >> > the > >> > > generic support. > >> > > > >> > > Best regards, > >> > > > >> > > Martijn > >> > > > >> > > > >> > > > >> > > Op do 14 jul. 2022 om 05:30 schreef bo zhaobo < > >> > bzhaojyathousa...@gmail.com > >> > > >: > >> > > > >> > > > Hi Martijn, > >> > > > > >> > > > Thank you for your comments. I will answer the questions one by > one. > >> > > > > >> > > > "" > >> > > > * Regarding the motivation, it mentions that the development trend > >> is > >> > > that > >> > > > Flink supports both batch and stream processing. I think the > vision > >> and > >> > > > trend is that we have unified batch- and stream processing. What > I'm > >> > > > missing is the vision on what's the impact for customized > Kubernetes > >> > > > schedulers on stream processing. Could there be some elaboration > on > >> > that? > >> > > > "" > >> > > > > >> > > > >> > >> > > > > >> > > > We very much agree with you and the dev trend that Flink supports > >> both > >> > > > batch and stream processing. Actually, using the K8S customized > >> > scheduler > >> > > > is beneficial for streaming scenarios too, such as avoiding > resource > >> > > > deadlock and other problems, for example, the remaining resources > in > >> > the > >> > > > K8S cluster are only enough for one job running, but we submitted > >> two. > >> > At > >> > > > this time, both jobs will be prevented and hang from requesting > >> > resources > >> > > > at the same time when using the default K8S scheduler, but in this > >> > case, > >> > > > the customized scheduler Volcano won’t schedule overcommit pods if > >> the > >> > > idle > >> > > > can not fit all following pods setup. So the benefits mentioned in > >> FLIP > >> > > are > >> > > > not only for batch jobs. In fact, the said 4 scheduling > capabilities > >> > > > mentioned in FLIP are all required for stream processing. YARN has > >> some > >> > > of > >> > > > those scheduling features too, such as priority scheduling, > min/max > >> > > > resource constraint and etc... > >> > > > > >> > > > "" > >> > > > * While the FLIP talks about customized schedulers, it focuses on > >> > > Volcano. > >> > > > Why is the choice made to only focus on Volcano and not on other > >> > > schedulers > >> > > > like Apache YuniKorn? Can we not also provide an implementation > for > >> > > > YuniKorn at the same time? Spark did the same with SPARK-36057 [1] > >> > > > "" > >> > > > > >> > > > >> > >> > > > > >> > > > Let's make it more clear about this. The FLIP consists of two > parts: > >> > > > 1. Introducing Flink K8S supports the customized scheduler plugin > >> > > > mechanism. This aspect is a general consideration. > >> > > > 2. Introducing ONE reference implementation for the customized > >> > scheduler, > >> > > > volcano is just one of them, if other schedulers or people are > >> > > interested, > >> > > > the integration of other schedulers can also be easily completed. > >> > > > > >> > > > "" > >> > > > * We still have quite a lot of tech debt on testing for Kubernetes > >> > [2]. I > >> > > > think that this FLIP would be a great improvement for Flink, but I > >> am > >> > > > worried that we will add more tech debt to those tests. Can we > >> somehow > >> > > > improve this situation? > >> > > > "" > >> > > > > >> > > > >> > >> > > > > >> > > > Yeah, We will pay attention to the test problems, which are > related > >> to > >> > > > Flink K8S and we are happy to improve it. ;-) > >> > > > > >> > > > BR, > >> > > > > >> > > > Bo Zhao > >> > > > > >> > > > Martijn Visser <martijnvis...@apache.org> 于2022年7月13日周三 15:19写道： > >> > > > > >> > > > > Hi all, > >> > > > > > >> > > > > Thanks for the FLIP. I have a couple of remarks/questions: > >> > > > > > >> > > > > * Regarding the motivation, it mentions that the development > >> trend is > >> > > > that > >> > > > > Flink supports both batch and stream processing. I think the > >> vision > >> > and > >> > > > > trend is that we have unified batch- and stream processing. What > >> I'm > >> > > > > missing is the vision on what's the impact for customized > >> Kubernetes > >> > > > > schedulers on stream processing. Could there be some elaboration > >> on > >> > > that? > >> > > > > * While the FLIP talks about customized schedulers, it focuses > on > >> > > > Volcano. > >> > > > > Why is the choice made to only focus on Volcano and not on other > >> > > > schedulers > >> > > > > like Apache YuniKorn? Can we not also provide an implementation > >> for > >> > > > > YuniKorn at the same time? Spark did the same with SPARK-36057 > [1] > >> > > > > * We still have quite a lot of tech debt on testing for > Kubernetes > >> > > [2]. I > >> > > > > think that this FLIP would be a great improvement for Flink, but > >> I am > >> > > > > worried that we will add more tech debt to those tests. Can we > >> > somehow > >> > > > > improve this situation? > >> > > > > > >> > > > > Best regards, > >> > > > > > >> > > > > Martijn > >> > > > > > >> > > > > [1] https://issues.apache.org/jira/browse/SPARK-36057 > >> > > > > [2] https://issues.apache.org/jira/browse/FLINK-20392 > >> > > > > > >> > > > > Op wo 13 jul. 2022 om 04:11 schreef 王正 <cswangzh...@gmail.com>: > >> > > > > > >> > > > > > +1 > >> > > > > > > >> > > > > > On 2022/07/07 01:15:13 bo zhaobo wrote: > >> > > > > > > Hi, all. > >> > > > > > > > >> > > > > > > I would like to raise a discussion in Flink dev ML about > >> Support > >> > > > > > Customized > >> > > > > > > Kubernetes Schedulers. > >> > > > > > > Currentlly, Kubernetes becomes more and more polular for > Flink > >> > > > Cluster > >> > > > > > > deployment, and its ability is better, especially, it > supports > >> > > > > > customized > >> > > > > > > scheduling. > >> > > > > > > Essentially, in high-performance workloads, we need to apply > >> new > >> > > > > > scheduling > >> > > > > > > policies for meeting the new requirements. And now Flink > >> native > >> > > > > > Kubernetes > >> > > > > > > solution is using Kubernetes default scheduler to work with > >> all > >> > > > > > scenarios, > >> > > > > > > the default scheduling policy might be difficult to apply in > >> some > >> > > > > extreme > >> > > > > > > cases, so > >> > > > > > > we need to improve the Flink Kubernetes for coupling those > >> > > Kubernetes > >> > > > > > > customized schedulers with Flink native Kubernetes, > provides a > >> > way > >> > > > for > >> > > > > > Flink > >> > > > > > > administrators or users to use/apply their Flink Clusters on > >> > > > Kubernetes > >> > > > > > > more flexibility. > >> > > > > > > > >> > > > > > > The proposal will introduce the customized K8S schdulers > >> plugin > >> > > > > mechanism > >> > > > > > > and a reference implementation 'Volcano' in Flink. More > >> details > >> > see > >> > > > > [1]. > >> > > > > > > > >> > > > > > > Looking forward to your feedback. > >> > > > > > > > >> > > > > > > [1] > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > BR > >> > > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
