Hi All, The @Public -> @PublicEvolving proposed by Xintong is a great idea. Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2 minor release)--> @Public --> @deprecated --(1 major release)--> @PublicRetired. It will provide a lot of flexibility without breaking any rules we had. @Public APIs are allowed to change between major releases. Changing annotations is acceptable and provides additional tolerance i.e. user-friendliness, since the APIs themself are not changed.
I had similar thoughts when I was facing those issues. I want to move one step further and suggest introducing one more annotation @Retired. Not like the @PublicRetired which is a compromise of downgrading @Public to @PublicEvolving. As I mentioned earlier in my reply, Java standard @deprecated should be used in the early stage of the deprecation process and doesn't really meet our requirement. Since Java does not allow us to extend annotation, I think it would be feasible to have the new @Retired to help us monitor and manage the deprecation process, house cleaning, etc. Some ideas could be(open for discussion): @Retired: 1. There must be a replacement with functionality compatibility before APIs can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are mandatory (a FLIP is recommended). 2. APIs marked as @Retired will be removed after 1 minor release sharply (using ArchUnit to force it, needs further check whether it is possible). Devs who marked them as @Retired are responsible to remove them. 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are recommended. @Experimental -> @Retired and @Internal -> @Retired could also be used if it can increase user-friendliness or dev-friendliness, but not mandatory. 4. Some variables will be defined in @Retired to support the deprecation process management. Further extension is possible, since the annotation is built by us. Best regards, Jing On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <becket....@gmail.com> wrote: > Hi Xintong, > > Thanks for the explanation. Please see the replies inline below. > > I agree. And from my understanding, demoting a Public API is also a kind of > > such change, just like removing one, which can only happen with major > > version bumps. I'm not proposing to allow demoting Public APIs anytime, > but > > only in the case major version bumps happen before reaching the > > 2-minor-release migration period. Actually, demoting would be a weaker > > change compared to removing the API immediately upon major version bumps, > > in order to keep the commitment about the 2-minor-release migration > period. > > If the concern is that `@Public` -> `@PublicEvolving` sounds against > > conventions, we may introduce a new annotation if necessary, e.g., > > `@PublicRetiring`, to avoid confusions. > > As an end user who only uses Public APIs, if I don't change my code at all, > my expectation is the following: > 1. Upgrading from 1.x to 2.x may have issues. > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all > the 2.x versions. > Actually I think there are some dependency version resolution policies out > there which picks the highest minor version when the dependencies pull in > multiple minor versions of the same jar, which may be broken if we remove > the API in minor releases. > > I'm not sure about this. Yes, it's completely "legal" that we bump up the > > major version whenever a breaking change is needed. However, this also > > weakens the value of the commitment that public APIs will stay stable > > within the major release series, as the series can end anytime. IMHO, > short > > major release series are not something "make the end users happy", but > > backdoors that allow us as the developers to make frequent breaking > > changes. On the contrary, with the demoting approach, we can still have > > longer major release series, while only allowing Public APIs deprecated > at > > the end of the previous major version to be removed in the next major > > version. > > I totally agree that frequent major version bumps are not ideal, but here > we are comparing it with a minor version bump which removes a Public API. > So the context is that we have already decided to remove this Public API > while keeping everything else backwards compatible. I think a major version > bump is a commonly understood signal here, compared with a minor version > change. From end users' perspective, for those who are not impacted, in > this case upgrading a major version is not necessarily more involved than > upgrading a minor version - both should be as smooth as a dependency > version change. For those who are impacted, they will lose the Public API > anyways and a major version bump ensures there is no surprise. > > Thanks, > > Jiangjie (Becket) Qin > > On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > Public API is a well defined common concept, and one of its > >> convention is that it only changes with a major version change. > >> > > > > I agree. And from my understanding, demoting a Public API is also a kind > > of such change, just like removing one, which can only happen with major > > version bumps. I'm not proposing to allow demoting Public APIs anytime, > but > > only in the case major version bumps happen before reaching the > > 2-minor-release migration period. Actually, demoting would be a weaker > > change compared to removing the API immediately upon major version bumps, > > in order to keep the commitment about the 2-minor-release migration > period. > > If the concern is that `@Public` -> `@PublicEvolving` sounds against > > conventions, we may introduce a new annotation if necessary, e.g., > > `@PublicRetiring`, to avoid confusions. > > > > But it should be > >> completely OK to bump up the major version if we really want to get rid > of > >> a public API, right? > >> > > > > I'm not sure about this. Yes, it's completely "legal" that we bump up the > > major version whenever a breaking change is needed. However, this also > > weakens the value of the commitment that public APIs will stay stable > > within the major release series, as the series can end anytime. IMHO, > short > > major release series are not something "make the end users happy", but > > backdoors that allow us as the developers to make frequent breaking > > changes. On the contrary, with the demoting approach, we can still have > > longer major release series, while only allowing Public APIs deprecated > at > > the end of the previous major version to be removed in the next major > > version. > > > > Given our track record I would prefer a regular cycle (1-2 years) to > >> force us to think about this whole topic, and not put it again to the > >> wayside and giving us (and users) a clear expectation on when breaking > >> changes can be made. > >> > > > > +1. I personally think 2-3 years would be a good time for new major > > versions, or longer if there's no breaking changes needed. That makes 1-2 > > year a perfect time to revisit the topic, while leaving us more time to > > prepare the major release if needed. > > > > Best, > > > > Xintong > > > > > > > > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <ches...@apache.org> > > wrote: > > > >> On 13/06/2023 17:26, Becket Qin wrote: > >> > It would be valuable if we can avoid releasing minor versions for > >> previous > >> > major versions. > >> > >> On paper, /absolutely /agree, but I'm not sure how viable that is in > >> practice. > >> > >> On the current 2.0 agenda is potentially dropping support for Java 8/11, > >> which may very well be a problem for our current users. > >> > >> > >> On 13/06/2023 17:26, Becket Qin wrote: > >> > Thanks for the feedback and sorry for the confusion about Public API > >> > deprecation. I just noticed that there was a mistake in the NOTES part > >> for > >> > Public API due to a copy-paste error... I just fixed it. > >> I'm very relieved to hear that. Glad to hear that we are on the same > >> page on that note. > >> > >> > >> On 15/06/2023 15:20, Becket Qin wrote: > >> > But it should be > >> > completely OK to bump up the major version if we really want to get > rid > >> of > >> > a public API, right? > >> > >> Technically yes, but look at how long it took to get us to 2.0. ;) > >> > >> There's a separate discussion to be had on the cadence of major releases > >> going forward, and there seem to be different opinions on that. > >> > >> If we take the Kafka example of 2 minor releases between major ones, > >> that for us means that users have to potentially deal with breaking > >> changes every 6 months, which seems like a lot. > >> > >> Given our track record I would prefer a regular cycle (1-2 years) to > >> force us to think about this whole topic, and not put it again to the > >> wayside and giving us (and users) a clear expectation on when breaking > >> changes can be made. > >> > >> But again, maybe this should be in a separate thread. > >> > >> On 14/06/2023 11:37, Becket Qin wrote: > >> > Do you have an example of behavioral change in mind? Not sure I fully > >> > understand the concern for behavioral change here. > >> > >> This could be a lot of things. It can be performance in certain > >> edge-cases, a bug fix that users (maybe unknowingly) relied upon > >> (https://xkcd.com/1172/), a semantic change to some API. > >> > >> For a concrete example, consider the job submission. A few releases back > >> we made changes such that the initialization of the job master happens > >> asynchronously. > >> This meant the job submission call returns sooner, and the job state > >> enum was extended to cover this state. > >> API-wise we consider this a compatible change, but the observed behavior > >> may be different. > >> > >> Metrics are another example; I believe over time we changed what some > >> metrics returned a few times. > >> > > >
