I'm noticing some pushback on the idea of pre-emptively proposing any kind of breaking upgrade that would necessitate cutting a 3.0 release. I do understand the concern about introducing a breaking change... For a distributed messaging application like Pulsar, if clients needed to be simultaneously upgraded with brokers, that could be extremely difficult or infeasible for companies to coordinate without treating it like a migration to a new technology.
At the same time, do we want to be completely closed to the possibility that a breaking change could be required at some point in the future? If a circumstance like that appears, those are the kinds of situations that can lead to a fork. Are there certain kinds of breaking changes that are more acceptable than others? Also, if the forward looking plan is to never introduce breaking changes, when *would* we ever cut a Pulsar 3.x release? Do we have any criteria on what kinds of changes would necessitate cutting a new major release but would still be considered acceptable by the community? -- Devin Bost Sent from mobile Cell: 801-400-4602 On Sat, Oct 8, 2022, 2:14 PM Rajan Dhabalia <rdhaba...@apache.org> wrote: > This sounds like the current state of Apache Pulsar has a lot of issues and > it requires fundamental design changes to make it promising which is > definitely not true and I disagree with it. And I would be careful > comparing with Kafka as I still don't think the Kafka release has anything > to do with Pulsar's improvement. I would still recommend to list down all > the changes at one place so we can bring everyone on the same page. discuss > as a community and we make sure existing usecases continue using Pulsar and > not try to find Pulsar alternatives with incorrect disruption impression > and efforts they might have to put to upgrade or maintain pulsar. > > Thanks, > Rajan > > On Fri, Oct 7, 2022 at 7:49 PM Lari Hotari <lhot...@apache.org> wrote: > > > We could all have our own favorite names for this work. :) > > > > There's advice that you should disrupt yourself before someone disrupts > > you. > > Shouldn't we follow that advice for Apache Pulsar? We can disrupt Pulsar > > together with our Apache hats on. The catch is that since we are doing > > this, we will be able to learn and improve Pulsar so that we stay ahead > of > > competition. Pulsar was long ways ahead of competition for so many years, > > but Kafka is finally catching up. Did Kafka surpass Pulsar in some > aspects > > with the recent 3.3 release, where Kraft became GA? That's a question > that > > many might be asking. Why wouldn't we rev up Pulsar's engine and show the > > tail lights to Kafka? > > > > We don't have to have deadlines or any restrictions like that right now. > > The sky's the limit. > > Linus Torvalds has written a book called "Just for fun". I got my copy of > > this book signed by Linus himself in year 2000 at an event that the book > > publisher had organized in Finland. > > > > What if we did this "just for fun"? The intention could also be to beat > > Kafka, but that could be a boring goal for many. What if we could unleash > > some talent that is among us and hasn't had a chance to show its full > > potential? Opensource is about joy. It is about welcoming everyone to > join. > > Opensource should be egoless, although we must all admit that we don't > > succeed in that aspect. We must fight our biases. > > > > Jarek Potiuk explains the importance of being welcoming for success at > > Apache, in a 3-minute YouTube interview: > > https://www.youtube.com/watch?v=Dx5kQnVFo7E > > This interview is about Jarek's blog post "Success at Apache: Welcoming > > communities strengthens the Apache way": > > > > > https://news.apache.org/foundation/entry/success-at-apache-welcoming-communities > > I was pleased to meet Jarek at ApacheCon among so many other welcoming > > personalities of the Apache community and the Apache Pulsar community. > > > > Goals have to be ambitious. What if we set the bar really high? > > Apache Pulsar with 10 million topics in a cluster? > > Why not go up to 100 million topics? > > Just for fun. :) > > > > -Lari > > > > On 2022/10/07 22:53:59 Matteo Merli wrote: > > > I actually disagree with the term "Pulsar Next Gen", because I haven't > > > seen any proposal for which that would make sense to me to be called > > > so. > > > > > > Rajan: That's the whole point of breaking it down. If you accumulate > > > many "big" changes it introduces a lot of risk for instabilities and > > > incompatibilities. Breaking it down in multiple steps helps to see the > > > incremental changes and introduced them in a phased manner. > > > > > > > > > -- > > > Matteo Merli > > > <matteo.me...@gmail.com> > > > > > > On Fri, Oct 7, 2022 at 3:37 PM Rajan Dhabalia <rdhaba...@apache.org> > > wrote: > > > > > > > > Hi, > > > > > > > > Can we get the list of changes at one place which we are planning to > > get as > > > > part of 3.0. One thing I would like to see as a part of a major > > release, it > > > > CAN NOT impact existing usecases and users in any way which can force > > them > > > > to upgrade the client library. Applications using < 3.0 version > should > > > > continue getting all the client and server side enhancements and bug > > fixes. > > > > Failing to provide bug-fixes and features to client < 3.0 means we > are > > > > forcing them to upgrade client version by putting efforts to handle > all > > > > incompatibility. and that's something we should definitely prevent > > because > > > > Apache Pulsar is used by many large scale business usecases and we > > should > > > > accommodate and motivate them to continue using Apache Pulsar. > > > > I understand as a Pulsar community we should always try to progress > and > > > > build better but not at the cost of losing or reducing the Apache > > Pulsar > > > > community. > > > > > > > > Thanks, > > > > Rajan > > > > > > > > > > > > On Fri, Oct 7, 2022 at 12:41 PM Lari Hotari <lhot...@apache.org> > > wrote: > > > > > > > > > Thank you, Matteo. I agree that features should be delivered > > continuously > > > > > when that is possible. In this case, that might not apply. > > > > > > > > > > I also agree that calling this Pulsar 3.0 isn't necessarily aligned > > with > > > > > PIP-175 since an LTS release is when the major version is bumped. > > I'm fine > > > > > in calling this "Pulsar Next Gen" or something that calls out that > > this is > > > > > planning for making a major leap in Pulsar. > > > > > > > > > > There are several unresolved issues with PIP-45 and the Pulsar Load > > > > > balancer. The previously referred email threads contain a lot of > > context to > > > > > this. Resolving the issues efficiently will most likely result in > > breaking > > > > > changes, which will be the reason why it deserves a major version > > upgrade. > > > > > > > > > > We have discussed it before that it's crucial to have a path to > > migrate > > > > > users when there are breaking changes. This should be covered in > any > > of the > > > > > solutions that are introduced. Optimally, users of Pulsar would be > > able to > > > > > upgrade seamlessly to Pulsar Next Gen / Pulsar 3.0, but rolling > back > > might > > > > > not be directly supported. > > > > > > > > > > I am welcoming everyone to join this planning for the Apache Pulsar > > Next > > > > > Gen architecture. Please check the first email in this thread for > > details > > > > > of context, and start participating and contributing today. The > best > > way to > > > > > contribute is to participate in the email threads, since they > contain > > > > > details with better context. > > > > > > > > > > -Lari > > > > > > > > > > On 2022/10/07 18:03:00 Matteo Merli wrote: > > > > > > Given the past experiences and the discussions that already > > happened > > > > > > around "PIP-175: Extend time based release process", the idea is > to > > > > > > detach the 3.0 from "big-features" items or "incompatible > changes". > > > > > > > > > > > > The changes are going to get included as they are ready, within > > > > > > feature releases, and in a fully compatible way. We don't need to > > > > > > group them together and create unnecessary risk for the release > > > > > > schedule and the users. > > > > > > > > > > > > > > > > > > -- > > > > > > Matteo Merli > > > > > > <matteo.me...@gmail.com> > > > > > > > > > > > > On Fri, Oct 7, 2022 at 10:47 AM Lari Hotari <lhot...@apache.org> > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Greeting from ApacheCon North America 2022 from New Orleans! > > > > > > > We had a great conference with a dedicated Pulsar track. Thanks > > to all > > > > > presenters and everyone who attended. The talks weren't recorded, > > but the > > > > > slides will be later on posted on the conference website [1]. > > > > > > > > > > > > > > At ApacheCon there were several presentations about "the Apache > > way" > > > > > and what that means in practice. Based on that, we all know that no > > person > > > > > is nominated as the CTO of Apache Pulsar who decides on Pulsar 3.0 > > and when > > > > > that happens. It's us, the community, that serve that role > together. > > We > > > > > come together as individuals with the Apache hat on. Everyone is > > equal in > > > > > the community, regardless of whether they are contributors, > > committers or > > > > > PMC members. > > > > > > > We welcome everyone to participate. The small detail about > voting > > > > > shouldn't stop anyone from participating in any aspects of the > > planning for > > > > > the roadmap. > > > > > > > > > > > > > > I'll like to get the discussions going for Pulsar 3.0. We don't > > need a > > > > > separate decision to start planning that. Please correct me if I'm > > wrong or > > > > > if you have a different opinion. > > > > > > > > > > > > > > There are a few previous discussion threads that are related to > > Pulsar > > > > > 3.0 planning. > > > > > > > If you are interested in getting involved with Apache Pulsar > 3.0 > > > > > planning, I think that it makes sense for you to read these threads > > > > > carefully and reply to them. Please also suggest what you think > > makes sense. > > > > > > > > > > > > > > PIP-45 related: > > > > > https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj > > > > > > > Pulsar Load balancer / namespace bundle related: > > > > > > > > https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk > > > > > > > renaming topics: > > > > > > > > https://lists.apache.org/thread/vrr75rrh4trqlp14objh3snlfvmzdrp2 > > > > > > > backpressure: > > > > > https://lists.apache.org/thread/v7xy57qfzbhopoqbm75s6ng8xlhbr2q6 > > > > > > > > > > > > > > Long list of Metadata inconsistency issues by Zac Bentley: > > > > > > > https://github.com/apache/pulsar/issues/12555 > > > > > > > That would be a good starting point to understanding the data > > > > > inconsistency issues related to current PIP-45 design. Perhaps > those > > could > > > > > be addressed already before Pulsar 3.0? > > > > > > > > > > > > > > I'm looking forward to everyone's participation in the Apache > > Pulsar > > > > > 3.0 planning discussions. > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > 1 - https://www.apachecon.com/acna2022/schedule.html > > > > > > > > > > > > > > > > >
