Hi Ruibin, I think it may be hard to say which provider is alway more recommended than the other. The answer to your question depends heavily on your team's technical stack, your platform and your expectations on the new cluster. I *cannot* give you any advice but I just want share some observations of myself and hope it can provide more information for the discussion: 1. stability: Flink on YARN module and Hadoop ecosystem have developed for a longer period of time than Flink on K8S and K8S ecosystem. The codebase of Flink on YARN module is more stable and it could be easier to get relevant information about YARN provider. But in production environment, Flink on K8S is also used by lots of companies and I believe these 2 providers are all capable of processing most flink workloads. 2. popularity: K8s, or "cloud native", is getting more popular. 3. community support: no obvious difference but it is worthwhile to mention that flink kubernetes operator is in rapid development and maybe you can give it a try. 4. toolchains: these providers all have their solutions to log collection / monitoring with metrics / resource scaling. In summary, IMO, currently, for Flink's usage, there are no obvious advantages of one provider comparing with the other. I can see that most middle sized users are still using YARN for their big data. But K8s may make it easier to utilize your resources completely when you have other workloads(e.g. spark batch jobs) which can reduce the cost. When FLIP-271 <https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling> of Autoscaling is completed, maybe Flink on K8s will have some killer advantage.
Best, Biao Geng Ruibin Xing <xingro...@gmail.com> 于2022年12月19日周一 14:54写道： > Hi all, > > We are currently setting up a new Flink cluster and are trying to decide > on the best deployment method. As far as we know, Flink supports two > resource providers: YARN and Kubernetes. We are having difficulty > evaluating the pros and cons of each provider, particularly in terms of > stability, popularity, community support, and toolchains. > > Could someone please share their experience with these resource providers > and provide a recommendation for our new cluster? > > Best, Ruibin > >
