Hi Becket, Thanks for this FLIP! Having a deprecation process is really important. I understand some people’s concerns about the additional burden for project maintainers, but my personal experience with Kafka has been that it’s very liveable and that it’s well worth the benefit to users. In fact, users being able to confidently upgrade is also a benefit to maintainers, as we will get fewer questions from people stuck on very old versions.
One question: Can we consider only dropping deprecated APIs in major releases across the board? I understand that Experimental and PublicEvolving APIs are by definition less stable, but it seems like this should be reflected in the required deprecation period alone. I.e. that we must keep them around for at least zero or one minor release, not that we can drop them in a minor or patch release. The advantage of forbidding the removal of any API in minor or patch releases is that users will get a strong guarantee that they can bump the minor or patch version and still be able to compile, or even just re-link and know that they won’t face “MethodDef” exceptions at run time. This is a binary guarantee: if we allow removing even Experimental APIs outside of major releases, users can no longer confidently upgrade. Aside from that, I’d share my 2 cents on a couple of points: * I’d use the official Deprecated annotation instead of introducing our own flavor (Retired, etc), since Deprecated is well integrated into build tools and IDEs. * I wouldn’t worry about a demotion process in this FLIP; it seems orthogonal, and something that should probably be taken case-by-case anyway. * Aside from deprecation and removal, there have been some discussions about how to evolve APIs and behavior in compatible ways. This is somewhat of an art, and if folks haven’t wrestled with it before, it’ll take some time to become good at it. I feel like this topic should also be orthogonal to this FLIP, but FWIW, my suggestion would be to adopt a simple policy not to break existing user programs, and leave the “how” up to implementers and reviewers. Thanks again, John On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: > Hi All, > > The @Public -> @PublicEvolving proposed by Xintong is a great idea. > Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2 > minor release)--> @Public --> @deprecated --(1 major > release)--> @PublicRetired. It will provide a lot of flexibility without > breaking any rules we had. @Public APIs are allowed to change between major > releases. Changing annotations is acceptable and provides additional > tolerance i.e. user-friendliness, since the APIs themself are not changed. > > I had similar thoughts when I was facing those issues. I want to move one > step further and suggest introducing one more annotation @Retired. > > Not like the @PublicRetired which is a compromise of downgrading @Public to > @PublicEvolving. As I mentioned earlier in my reply, Java standard > @deprecated should be used in the early stage of the deprecation process > and doesn't really meet our requirement. Since Java does not allow us to > extend annotation, I think it would be feasible to have the new @Retired to > help us monitor and manage the deprecation process, house cleaning, etc. > > Some ideas could be(open for discussion): > > @Retired: > > 1. There must be a replacement with functionality compatibility before APIs > can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are > mandatory (a FLIP is recommended). > 2. APIs marked as @Retired will be removed after 1 minor release sharply > (using ArchUnit to force it, needs further check whether it is possible). > Devs who marked them as @Retired are responsible to remove them. > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are > recommended. @Experimental -> @Retired and @Internal -> @Retired could also > be used if it can increase user-friendliness or dev-friendliness, but not > mandatory. > 4. Some variables will be defined in @Retired to support the deprecation > process management. Further extension is possible, since the annotation is > built by us. > > > Best regards, > Jing > > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <becket....@gmail.com> wrote: > >> Hi Xintong, >> >> Thanks for the explanation. Please see the replies inline below. >> >> I agree. And from my understanding, demoting a Public API is also a kind of >> > such change, just like removing one, which can only happen with major >> > version bumps. I'm not proposing to allow demoting Public APIs anytime, >> but >> > only in the case major version bumps happen before reaching the >> > 2-minor-release migration period. Actually, demoting would be a weaker >> > change compared to removing the API immediately upon major version bumps, >> > in order to keep the commitment about the 2-minor-release migration >> period. >> > If the concern is that `@Public` -> `@PublicEvolving` sounds against >> > conventions, we may introduce a new annotation if necessary, e.g., >> > `@PublicRetiring`, to avoid confusions. >> >> As an end user who only uses Public APIs, if I don't change my code at all, >> my expectation is the following: >> 1. Upgrading from 1.x to 2.x may have issues. >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all >> the 2.x versions. >> Actually I think there are some dependency version resolution policies out >> there which picks the highest minor version when the dependencies pull in >> multiple minor versions of the same jar, which may be broken if we remove >> the API in minor releases. >> >> I'm not sure about this. Yes, it's completely "legal" that we bump up the >> > major version whenever a breaking change is needed. However, this also >> > weakens the value of the commitment that public APIs will stay stable >> > within the major release series, as the series can end anytime. IMHO, >> short >> > major release series are not something "make the end users happy", but >> > backdoors that allow us as the developers to make frequent breaking >> > changes. On the contrary, with the demoting approach, we can still have >> > longer major release series, while only allowing Public APIs deprecated >> at >> > the end of the previous major version to be removed in the next major >> > version. >> >> I totally agree that frequent major version bumps are not ideal, but here >> we are comparing it with a minor version bump which removes a Public API. >> So the context is that we have already decided to remove this Public API >> while keeping everything else backwards compatible. I think a major version >> bump is a commonly understood signal here, compared with a minor version >> change. From end users' perspective, for those who are not impacted, in >> this case upgrading a major version is not necessarily more involved than >> upgrading a minor version - both should be as smooth as a dependency >> version change. For those who are impacted, they will lose the Public API >> anyways and a major version bump ensures there is no surprise. >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <tonysong...@gmail.com> >> wrote: >> >> > Public API is a well defined common concept, and one of its >> >> convention is that it only changes with a major version change. >> >> >> > >> > I agree. And from my understanding, demoting a Public API is also a kind >> > of such change, just like removing one, which can only happen with major >> > version bumps. I'm not proposing to allow demoting Public APIs anytime, >> but >> > only in the case major version bumps happen before reaching the >> > 2-minor-release migration period. Actually, demoting would be a weaker >> > change compared to removing the API immediately upon major version bumps, >> > in order to keep the commitment about the 2-minor-release migration >> period. >> > If the concern is that `@Public` -> `@PublicEvolving` sounds against >> > conventions, we may introduce a new annotation if necessary, e.g., >> > `@PublicRetiring`, to avoid confusions. >> > >> > But it should be >> >> completely OK to bump up the major version if we really want to get rid >> of >> >> a public API, right? >> >> >> > >> > I'm not sure about this. Yes, it's completely "legal" that we bump up the >> > major version whenever a breaking change is needed. However, this also >> > weakens the value of the commitment that public APIs will stay stable >> > within the major release series, as the series can end anytime. IMHO, >> short >> > major release series are not something "make the end users happy", but >> > backdoors that allow us as the developers to make frequent breaking >> > changes. On the contrary, with the demoting approach, we can still have >> > longer major release series, while only allowing Public APIs deprecated >> at >> > the end of the previous major version to be removed in the next major >> > version. >> > >> > Given our track record I would prefer a regular cycle (1-2 years) to >> >> force us to think about this whole topic, and not put it again to the >> >> wayside and giving us (and users) a clear expectation on when breaking >> >> changes can be made. >> >> >> > >> > +1. I personally think 2-3 years would be a good time for new major >> > versions, or longer if there's no breaking changes needed. That makes 1-2 >> > year a perfect time to revisit the topic, while leaving us more time to >> > prepare the major release if needed. >> > >> > Best, >> > >> > Xintong >> > >> > >> > >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <ches...@apache.org> >> > wrote: >> > >> >> On 13/06/2023 17:26, Becket Qin wrote: >> >> > It would be valuable if we can avoid releasing minor versions for >> >> previous >> >> > major versions. >> >> >> >> On paper, /absolutely /agree, but I'm not sure how viable that is in >> >> practice. >> >> >> >> On the current 2.0 agenda is potentially dropping support for Java 8/11, >> >> which may very well be a problem for our current users. >> >> >> >> >> >> On 13/06/2023 17:26, Becket Qin wrote: >> >> > Thanks for the feedback and sorry for the confusion about Public API >> >> > deprecation. I just noticed that there was a mistake in the NOTES part >> >> for >> >> > Public API due to a copy-paste error... I just fixed it. >> >> I'm very relieved to hear that. Glad to hear that we are on the same >> >> page on that note. >> >> >> >> >> >> On 15/06/2023 15:20, Becket Qin wrote: >> >> > But it should be >> >> > completely OK to bump up the major version if we really want to get >> rid >> >> of >> >> > a public API, right? >> >> >> >> Technically yes, but look at how long it took to get us to 2.0. ;) >> >> >> >> There's a separate discussion to be had on the cadence of major releases >> >> going forward, and there seem to be different opinions on that. >> >> >> >> If we take the Kafka example of 2 minor releases between major ones, >> >> that for us means that users have to potentially deal with breaking >> >> changes every 6 months, which seems like a lot. >> >> >> >> Given our track record I would prefer a regular cycle (1-2 years) to >> >> force us to think about this whole topic, and not put it again to the >> >> wayside and giving us (and users) a clear expectation on when breaking >> >> changes can be made. >> >> >> >> But again, maybe this should be in a separate thread. >> >> >> >> On 14/06/2023 11:37, Becket Qin wrote: >> >> > Do you have an example of behavioral change in mind? Not sure I fully >> >> > understand the concern for behavioral change here. >> >> >> >> This could be a lot of things. It can be performance in certain >> >> edge-cases, a bug fix that users (maybe unknowingly) relied upon >> >> (https://xkcd.com/1172/), a semantic change to some API. >> >> >> >> For a concrete example, consider the job submission. A few releases back >> >> we made changes such that the initialization of the job master happens >> >> asynchronously. >> >> This meant the job submission call returns sooner, and the job state >> >> enum was extended to cover this state. >> >> API-wise we consider this a compatible change, but the observed behavior >> >> may be different. >> >> >> >> Metrics are another example; I believe over time we changed what some >> >> metrics returned a few times. >> >> >> > >>
