+1 for officially deprecating this component for the 1.13 release. Cheers, Till
On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kna...@apache.org> wrote: > Hi Matthias, > > Thank you for following up on this. +1 to officially deprecate Mesos in > the code and documentation, too. It will be confusing for users if this > diverges from the roadmap. > > Cheers, > > Konstantin > > On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <matth...@ververica.com> > wrote: > >> Hi everyone, >> considering the upcoming release of Flink 1.13, I wanted to revive the >> discussion about the Mesos support ones more. Mesos is also already listed >> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align >> the >> documentation accordingly to make it more explicit? >> >> What do you think? >> >> Best, >> Matthias >> >> [1] https://flink.apache.org/roadmap.html#feature-radar >> >> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >> > Hi Oleksandr, >> > >> > yes you are right. The biggest problem is at the moment the lack of test >> > coverage and thereby confidence to make changes. We have some e2e tests >> > which you can find here [1]. These tests are, however, quite coarse >> grained >> > and are missing a lot of cases. One idea would be to add a Mesos e2e >> test >> > based on Flink's end-to-end test framework [2]. I think what needs to be >> > done there is to add a Mesos resource and a way to submit jobs to a >> Mesos >> > cluster to write e2e tests. >> > >> > [1] https://github.com/apache/flink/tree/master/flink-jepsen >> > [2] >> > >> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common >> > >> > Cheers, >> > Till >> > >> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi < >> > o.nitavs...@criteo.com> wrote: >> > >> >> Hello Xintong, >> >> >> >> Thanks for the insights and support. >> >> >> >> Browsing the Mesos backlog and didn't identify anything critical, which >> >> is left there. >> >> >> >> I see that there are were quite a lot of contributions to the Flink >> Mesos >> >> in the recent version: >> >> https://github.com/apache/flink/commits/master/flink-mesos. >> >> We plan to validate the current Flink master (or release 1.12 branch) >> our >> >> Mesos setup. In case of any issues, we will try to propose changes. >> >> My feeling is that our test results shouldn't affect the Flink 1.12 >> >> release cycle. And if any potential commits will land into the 1.12.1 >> it >> >> should be totally fine. >> >> >> >> In the future, we would be glad to help you guys with any >> >> maintenance-related questions. One of the highest priorities around >> this >> >> component seems to be the development of the full e2e test. >> >> >> >> Kind Regards >> >> Oleksandr Nitavskyi >> >> ________________________________ >> >> From: Xintong Song <tonysong...@gmail.com> >> >> Sent: Tuesday, October 27, 2020 7:14 AM >> >> To: dev <d...@flink.apache.org>; user <user@flink.apache.org> >> >> Cc: Piyush Narang <p.nar...@criteo.com> >> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support >> >> >> >> Hi Piyush, >> >> >> >> Thanks a lot for sharing the information. It would be a great relief >> that >> >> you are good with Flink on Mesos as is. >> >> >> >> As for the jira issues, I believe the most essential ones should have >> >> already been resolved. You may find some remaining open issues here >> [1], >> >> but not all of them are necessary if we decide to keep Flink on Mesos >> as is. >> >> >> >> At the moment and in the short future, I think helps are mostly needed >> on >> >> testing the upcoming release 1.12 with Mesos use cases. The community >> is >> >> currently actively preparing the new release, and hopefully we could >> come >> >> up with a release candidate early next month. It would be greatly >> >> appreciated if you fork as experienced Flink on Mesos users can help >> with >> >> verifying the release candidates. >> >> >> >> >> >> Thank you~ >> >> >> >> Xintong Song >> >> >> >> [1] >> >> >> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open >> >> < >> >> >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0 >> >> > >> >> >> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.nar...@criteo.com >> >> <mailto:p.nar...@criteo.com>> wrote: >> >> >> >> Hi Xintong, >> >> >> >> >> >> >> >> Do you have any jiras that cover any of the items on 1 or 2? I can >> reach >> >> out to folks internally and see if I can get some folks to commit to >> >> helping out. >> >> >> >> >> >> >> >> To cover the other qs: >> >> >> >> * Yes, we’ve not got a plan at the moment to get off Mesos. We use >> >> Yarn for some our Flink workloads when we can. Mesos is only used when >> we >> >> need streaming capabilities in our WW dcs (as our Yarn is centralized >> in >> >> one DC) >> >> * We’re currently on Flink 1.9 (old planner). We have a plan to >> bump >> >> to 1.11 / 1.12 this quarter. >> >> * We typically upgrade once every 6 months to a year (not every >> >> release). We’d like to speed up the cadence but we’re not there yet. >> >> * We’d largely be good with keeping Flink on Mesos as-is and >> >> functional while missing out on some of the newer features. We >> understand >> >> the pain on the communities side and we can take on the work if we see >> some >> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put >> in >> >> the request to port it over. >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> -- Piyush >> >> >> >> >> >> >> >> >> >> >> >> From: Xintong Song <tonysong...@gmail.com<mailto:tonysong...@gmail.com >> >> >> >> Date: Sunday, October 25, 2020 at 10:57 PM >> >> To: dev <d...@flink.apache.org<mailto:d...@flink.apache.org>>, user < >> >> user@flink.apache.org<mailto:user@flink.apache.org>> >> >> Cc: Lasse Nedergaard <lassenedergaardfl...@gmail.com<mailto: >> >> lassenedergaardfl...@gmail.com>>, <p.nar...@criteo.com<mailto: >> >> p.nar...@criteo.com>> >> >> Subject: Re: [SURVEY] Remove Mesos support >> >> >> >> >> >> >> >> Thanks for sharing the information with us, Piyush an Lasse. >> >> >> >> >> >> >> >> @Piyush >> >> >> >> >> >> >> >> Thanks for offering the help. IMO, there are currently several problems >> >> that make supporting Flink on Mesos challenging for us. >> >> >> >> 1. Lack of Mesos experts. AFAIK, there are very few people (if not >> >> none) among the active contributors in this community that are familiar >> >> with Mesos and can help with development on this component. >> >> 2. Absence of tests. Mesos does not provide a testing cluster, like >> >> `MiniYARNCluster`, making it hard to test interactions between Flink >> and >> >> Mesos. We have only a few very simple e2e tests running on Mesos >> deployed >> >> in a docker, covering the most fundamental workflows. We are not sure >> how >> >> well those tests work, especially against some potential corner cases. >> >> 3. Divergence from other deployment. Because of 1 and 2, the new >> >> efforts (features, maintenance, refactors) tend to exclude Mesos if >> >> possible. When the new efforts have to touch the Mesos related >> components >> >> (e.g., changes to the common resource manager interfaces), we have to >> be >> >> very careful and make as few changes as possible, to avoid accidentally >> >> breaking anything that we are not familiar with. As a result, the >> component >> >> diverges a lot from other deployment components (K8s/Yarn), which >> makes it >> >> harder to maintain. >> >> >> >> It would be greatly appreciated if you can help with either of the >> above >> >> issues. >> >> >> >> >> >> >> >> Additionally, I have a few questions concerning your use cases at >> Criteo. >> >> IIUC, you are going to stay on Mesos in the foreseeable future, while >> >> keeping the Flink version up-to-date? What Flink version are you >> currently >> >> using? How often do you upgrade (e.g., every release)? Would you be >> good >> >> with keeping the Flink on Mesos component as it is (means that >> deployment >> >> and resource management improvements may not be ported to Mesos), while >> >> keeping other components up-to-date (e.g., improvements from >> programming >> >> APIs, operators, state backens, etc.)? >> >> >> >> >> >> >> >> Thank you~ >> >> >> >> Xintong Song >> >> >> >> >> >> >> >> >> >> >> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard < >> >> lassenedergaardfl...@gmail.com<mailto:lassenedergaardfl...@gmail.com>> >> >> wrote: >> >> >> >> Hi >> >> >> >> >> >> >> >> At Trackunit We have been using Mesos for long time but have now moved >> to >> >> k8s. >> >> >> >> Med venlig hilsen / Best regards >> >> >> >> Lasse Nedergaard >> >> >> >> >> >> >> >> >> >> >> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetz...@apache.org >> >> <mailto:rmetz...@apache.org>>: >> >> >> >> >> >> >> >> Hey Piyush, >> >> >> >> thanks a lot for raising this concern. I believe we should keep Mesos >> in >> >> Flink then in the foreseeable future. >> >> >> >> Your offer to help is much appreciated. We'll let you know once there >> is >> >> something. >> >> >> >> >> >> >> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.nar...@criteo.com >> >> <mailto:p.nar...@criteo.com>> wrote: >> >> >> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be able >> >> to find folks who would be excited to contribute / help in any way. >> >> >> >> -- Piyush >> >> >> >> >> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kklou...@gmail.com<mailto: >> >> kklou...@gmail.com>> wrote: >> >> >> >> Thanks Piyush for the message. >> >> After this, I revoke my +1. I agree with the previous opinions >> that we >> >> cannot drop code that is actively used by users, especially if it >> >> something that deep in the stack as support for cluster management >> >> framework. >> >> >> >> Cheers, >> >> Kostas >> >> >> >> On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.nar...@criteo.com >> >> <mailto:p.nar...@criteo.com>> wrote: >> >> > >> >> > Hi folks, >> >> > >> >> > >> >> > >> >> > We at Criteo are active users of the Flink on Mesos resource >> >> management component. We are pretty heavy users of Mesos for scheduling >> >> workloads on our edge datacenters and we do want to continue to be >> able to >> >> run some of our Flink topologies (to compute machine learning short >> term >> >> features) on those DCs. If possible our vote would be not to drop Mesos >> >> support as that will tie us to an old release / have to maintain a >> fork as >> >> we’re not planning to migrate off Mesos anytime soon. Is the burden >> >> something that can be helped with by the community? (Or are you >> referring >> >> to having to ensure PRs handle the Mesos piece as well when they touch >> the >> >> resource managers?) >> >> > >> >> > >> >> > >> >> > Thanks, >> >> > >> >> > >> >> > >> >> > -- Piyush >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > From: Till Rohrmann <trohrm...@apache.org<mailto: >> >> trohrm...@apache.org>> >> >> > Date: Friday, October 23, 2020 at 8:19 AM >> >> > To: Xintong Song <tonysong...@gmail.com<mailto: >> >> tonysong...@gmail.com>> >> >> > Cc: dev <d...@flink.apache.org<mailto:d...@flink.apache.org>>, >> user < >> >> user@flink.apache.org<mailto:user@flink.apache.org>> >> >> > Subject: Re: [SURVEY] Remove Mesos support >> >> > >> >> > >> >> > >> >> > Thanks for starting this survey Robert! I second Konstantin and >> >> Xintong in the sense that our Mesos user's opinions should matter most >> >> here. If our community is no longer using the Mesos integration, then I >> >> would be +1 for removing it in order to decrease the maintenance >> burden. >> >> > >> >> > >> >> > >> >> > Cheers, >> >> > >> >> > Till >> >> > >> >> > >> >> > >> >> > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song < >> tonysong...@gmail.com >> >> <mailto:tonysong...@gmail.com>> wrote: >> >> > >> >> > +1 for adding a warning in 1.12 about planning to remove Mesos >> >> support. >> >> > >> >> > >> >> > >> >> > With my developer hat on, removing the Mesos support would >> >> definitely reduce the maintaining overhead for the deployment and >> resource >> >> management related components. On the other hand, the Flink on Mesos >> users' >> >> voices definitely matter a lot for this community. Either way, it >> would be >> >> good to draw users attention to this discussion early. >> >> > >> >> > >> >> > >> >> > Thank you~ >> >> > >> >> > Xintong Song >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf < >> kna...@apache.org >> >> <mailto:kna...@apache.org>> wrote: >> >> > >> >> > Hi Robert, >> >> > >> >> > +1 to the plan you outlined. If we were to drop support in Flink >> >> 1.13+, we >> >> > would still support it in Flink 1.12- with bug fixes for some >> time >> >> so that >> >> > users have time to move on. >> >> > >> >> > It would certainly be very interesting to hear from current Flink >> >> on Mesos >> >> > users, on how they see the evolution of this part of the >> ecosystem. >> >> > >> >> > Best, >> >> > >> >> > Konstantin >> >> >> > >> > > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk >
