Sorry for the late jump in. +1 to keep the compatibility of @PublicEvolving between minor releases(x.y.a -> x.y.b), as a user I always think this as a bug-fix release, break the compatibility between minor releases may give users a surprise.
As the previous emails said, how and when will a @PublicEvolving become @Public, and I'm not sure if we can have a technical solution to keep such a rule. (In my opinion, check such things -- change @PublicEvolving to @Public -- manually may not so easy) Best Congxian Till Rohrmann <trohrm...@apache.org> 于2020年5月15日周五 下午9:18写道： > The vote thread can be found here > > https://lists.apache.org/thread.html/rc58099fb0e31d0eac951a7bbf7f8bda8b7b65c9ed0c04622f5333745%40%3Cdev.flink.apache.org%3E > . > > Cheers, > Till > > On Fri, May 15, 2020 at 3:03 PM Till Rohrmann <trohrm...@apache.org> > wrote: > > > I completely agree that there are many other aspect of our guarantees and > > processes around the @Public and @PublicEvolving classes which need to be > > discussed and properly defined. For the sake of keeping this discussion > > thread narrowly scoped, I would suggest to start a separate discussion > > about the following points (not exhaustive): > > > > - What should be annotated with @Public and @PublicEvolving? > > - Process for transforming @PublicEvolving into @Public; How to ensure > > that @PublicEvolving will eventually be promoted to @Public? > > - Process of retiring a @Public/@PublicEvolving API > > > > I will start a vote thread about the change I proposed here which is to > > ensure API and binary compatibility for @PublicEvolving classes between > > bugfix releases (x.y.z and x.y.u). > > > > Cheers, > > Till > > > > On Fri, May 15, 2020 at 6:33 AM Zhu Zhu <reed...@gmail.com> wrote: > > > >> +1 for "API + binary compatibility for @PublicEvolving classes for all > bug > >> fix > >> releases in a minor release (x.y.z is compatible to x.y.u)" > >> > >> This @PublicEnvolving would then be a hard limit to changes. > >> So it's important to rethink the policy towards using it, as Stephan > >> proposed. > >> > >> I think any Flink interfaces that are visible to users should be > >> explicitly > >> marked as @Public or @PublicEnvolving. > >> Any other interfaces should not be marked as @Public/@PublicEnvolving. > >> This would be essential for us to check whether we are breaking any user > >> faced interfaces unexpectedly. > >> The only exception would be the case that we had to expose a > method/class > >> due to implementation limitations, it should be explicitly marked it > >> as @Internal. > >> > >> Thanks, > >> Zhu Zhu > >> > >> Yun Tang <myas...@live.com> 于2020年5月15日周五 上午11:41写道： > >> > >> > +1 for this idea, and I also like Xintong's suggestion to make it > >> > explicitly when the @PublicEvolving API could upgrade to @Public API. > >> > If we have the rule to upgrade API stable level but not define the > clear > >> > timeline, I'm afraid not everyone have the enthusiasm to upgrade this. > >> > > >> > The minor suggestion is that I think two major release (which is x.y.0 > >> as > >> > Chesnay clarified) might be a bit quick. From the release history [1], > >> > Flink bump major version every 3 ~ 6 months and two major release gap > >> > could only be at least half a year. > >> > I think half a year might be a bit too frequent for users to collect > >> > enough feedbacks, and upgrading API stable level every 3 major > versions > >> > should be better. > >> > > >> > [1] https://flink.apache.org/downloads.html#flink > >> > > >> > Best > >> > Yun Tang > >> > > >> > > >> > ________________________________ > >> > From: Xintong Song <tonysong...@gmail.com> > >> > Sent: Friday, May 15, 2020 11:04 > >> > To: dev <dev@flink.apache.org> > >> > Subject: Re: [DISCUSS] Stability guarantees for @PublicEvolving > classes > >> > > >> > ### Documentation on API compatibility policies > >> > > >> > Do we have any formal documentation about the API compatibility > >> policies? > >> > The only things I found are: > >> > > >> > - In the release announcement (take 1.10.0 as an example) [1]: > >> > "This version is API-compatible with previous 1.x releases for APIs > >> > annotated with the @Public annotation." > >> > - JavaDoc for Public [2] and PublicEvolving [3]. > >> > > >> > I think we might have a formal documentation, clearly state our > policies > >> > for API compatibility. > >> > > >> > - What does the annotations mean > >> > - In what circumstance would the APIs remain compatible / become > >> > incompatible > >> > - How do APIs retire (e.g., first deprecated then removed?) > >> > > >> > Maybe there is already such kind of documentation that I overlooked? > If > >> so, > >> > we probably want to make it more explicit and easy-to-find. > >> > > >> > ### @Public vs. @PublicEvolving for new things > >> > > >> > I share Stephan's concern that, with @PublicEvolving used for every > new > >> > feature and rarely upgraded to @Public, we are practically making no > >> > compatibility guarantee between minor versions (x.y.* / x.z.*). On the > >> > other hand, I think in many circumstances we do need some time to > >> collect > >> > feedbacks for new features before we have enough confidence to make > the > >> > commitment that our APIs are stable. Therefore, it makes more sense to > >> me > >> > to first make new features @PublicEvolving and then upgrade to @Public > >> in > >> > the next one or two releases (unless there's a good reason to further > >> > postpone it). > >> > > >> > I think the key point is how do we make sure the @PublicEvolving > >> features > >> > upgrade to @Public. Maybe we can add a parameter to indicate the > >> expected > >> > upgrading version. E.g., a new feature introduced in release 1.10.0 > >> might > >> > be annotated as @PublicEvolving("1.12.0"), indicating that it is > >> expected > >> > to be upgraded to @Public in release 1.12.0. We can check the > >> annotations > >> > against the version automatically, forcing to either upgrad the > feature > >> > to @Public or explicitly postpone it by modifying the annotation > >> parameter > >> > (if there's a good reason). > >> > > >> > Additionally, we can do the similar for deprecated features / APIs, > >> > reminding us to remove things annotated as @Deprecated at certain > time. > >> > > >> > Thank you~ > >> > > >> > Xintong Song > >> > > >> > > >> > [1] https://flink.apache.org/news/2020/02/11/release-1.10.0.html > >> > > >> > [2] > >> > > >> > > >> > https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/annotation/Public.html > >> > > >> > [3] > >> > > >> > > >> > https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/annotation/PublicEvolving.html > >> > > >> > > >> > > >> > > >> > On Thu, May 14, 2020 at 8:22 PM Stephan Ewen <se...@apache.org> > wrote: > >> > > >> > > I just want to throw in that we also need to rethink our policy > >> towards > >> > > using @PublicEvolving. > >> > > > >> > > We often introduce this easily (for every new feature) and rarely > >> (almost > >> > > never) upgrade it to @Public. This kind of leads the idea behind > >> stable > >> > API > >> > > guarantees ad absurdum. > >> > > > >> > > I would suggest that we make @PublicEvolving an exception that > needs a > >> > good > >> > > reason rather than for everything that is new (when we don't want to > >> be > >> > > bothered with thinking about compatibility). > >> > > > >> > > > >> > > On Thu, May 14, 2020 at 1:05 PM Xintong Song <tonysong...@gmail.com > > > >> > > wrote: > >> > > > >> > > > Thanks for the clarification. > >> > > > +1 for keeping the current guarantees for @Public. > >> > > > > >> > > > Thank you~ > >> > > > > >> > > > Xintong Song > >> > > > > >> > > > > >> > > > > >> > > > On Thu, May 14, 2020 at 6:07 PM Till Rohrmann < > trohrm...@apache.org > >> > > >> > > > wrote: > >> > > > > >> > > > > Sorry for the confusion. @Public classes are guaranteed to be > >> stable > >> > > > > between releases x.y.z and x.u.v (minor and bug fix release; > >> naming > >> > is > >> > > > > indeed a bit off here) and we can break it with major releases > >> (x.0.0 > >> > > and > >> > > > > y.0.0). > >> > > > > > >> > > > > @Tison I would like to make what to include in the public API, > >> hence > >> > > what > >> > > > > to annotate with @Public and @PublicEvolving, a separate > >> discussion. > >> > > > > > >> > > > > Cheers, > >> > > > > Till > >> > > > > > >> > > > > On Thu, May 14, 2020 at 11:48 AM tison <wander4...@gmail.com> > >> wrote: > >> > > > > > >> > > > >> Thanks for starting this discussion! > >> > > > >> > >> > > > >> I agree turn on japicmp on PublicEvolving among bugfix releases > >> is a > >> > > nit > >> > > > >> win. > >> > > > >> > >> > > > >> @Xintong Song <tonysong...@gmail.com> I think @Public > guarantee > >> is > >> > > good > >> > > > >> enough, the problem is a reachable 2.0 plan. > >> > > > >> > >> > > > >> My concern is more on classes that have no annotation but our > >> > > developers > >> > > > >> regard as "something that should be stable". Previously I was > >> > required > >> > > > to > >> > > > >> keep compatibility of ClusterClient & HighAvailabilityServices > >> > because > >> > > > >> they > >> > > > >> might be depended on by user. > >> > > > >> > >> > > > >> Best, > >> > > > >> tison. > >> > > > >> > >> > > > >> > >> > > > >> Dawid Wysakowicz <dwysakow...@apache.org> 于2020年5月14日周四 > >> 下午5:08写道： > >> > > > >> > >> > > > >> > I also like the proposal for keeping the binary compatibility > >> of > >> > > > >> > @PublicEvolving for bugfix releases. > >> > > > >> > > >> > > > >> > As for the @Public classes I think the current guarantees are > >> good > >> > > > >> enough. > >> > > > >> > > >> > > > >> > Best, > >> > > > >> > > >> > > > >> > Dawid > >> > > > >> > > >> > > > >> > On 14/05/2020 10:49, Jingsong Li wrote: > >> > > > >> > > Thanks Till for starting this discussion. > >> > > > >> > > > >> > > > >> > > +1 for enabling the japicmp-maven-plugin for > @PublicEvolving > >> for > >> > > bug > >> > > > >> fix > >> > > > >> > > releases. > >> > > > >> > > Bug fix should just be user imperceptible bug fix. Should > not > >> > > affect > >> > > > >> API > >> > > > >> > > and binary compatibility. > >> > > > >> > > > >> > > > >> > > And even PublicEvolving api change for "y" release, we > should > >> > > expose > >> > > > >> it > >> > > > >> > in > >> > > > >> > > dev mail list for discussing or a FLIP? > >> > > > >> > > > >> > > > >> > > BTW, public api can be changed by major releases? In > >> annotation > >> > > > >> comments: > >> > > > >> > > "Only major releases (1.0, 2.0, 3.0) can break interfaces > >> with > >> > > this > >> > > > >> > > annotation". > >> > > > >> > > > >> > > > >> > > Best, > >> > > > >> > > Jingsong Lee > >> > > > >> > > > >> > > > >> > > On Thu, May 14, 2020 at 4:30 PM Till Rohrmann < > >> > > trohrm...@apache.org > >> > > > > > >> > > > >> > wrote: > >> > > > >> > > > >> > > > >> > >> Dear community, > >> > > > >> > >> > >> > > > >> > >> in the latest 1.10.1 bug fix release I introduced a binary > >> > > > >> incompatible > >> > > > >> > >> change to a class which is annotated with @PublicEvolving > >> [1]. > >> > > > While > >> > > > >> > this > >> > > > >> > >> change is technically ok since we only provide API and > >> binary > >> > > > >> > compatibility > >> > > > >> > >> for @Public classes across releases, it raised the > question > >> > > whether > >> > > > >> we > >> > > > >> > >> can't do better. > >> > > > >> > >> > >> > > > >> > >> For our users it might be surprising and really annoying > >> that > >> > > they > >> > > > >> > cannot > >> > > > >> > >> simply upgrade to the latest bug fix release without > >> > recompiling > >> > > > the > >> > > > >> > >> program or even having to change the source code of an > >> > > > application. I > >> > > > >> > >> believe we would provide a much better experience if we > >> ensured > >> > > > that > >> > > > >> bug > >> > > > >> > >> fix releases maintain API and binary compatibility also > for > >> > > > >> > @PublicEvolving > >> > > > >> > >> classes. Hence my proposal would be to tighten the > stability > >> > > > >> guarantees > >> > > > >> > the > >> > > > >> > >> following way: > >> > > > >> > >> > >> > > > >> > >> * API + binary compatibility for @Public classes across > all > >> > > > releases > >> > > > >> > (x.y.z > >> > > > >> > >> is compatible to u.v.w) > >> > > > >> > >> * API + binary compatibility for @PublicEvolving classes > for > >> > all > >> > > > bug > >> > > > >> fix > >> > > > >> > >> releases in a minor release (x.y.z is compatible to x.y.u) > >> > > > >> > >> > >> > > > >> > >> This would entail that we can change @PublicEvolving > classes > >> > only > >> > > > >> across > >> > > > >> > >> minor/major releases. > >> > > > >> > >> > >> > > > >> > >> Practically this would mean that we enable the > >> > > japicmp-maven-plugin > >> > > > >> > >> for @PublicEvolving for bug fix releases. > >> > > > >> > >> > >> > > > >> > >> What do you think? > >> > > > >> > >> > >> > > > >> > >> [1] > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/r293768d13d08149d756e0bf91be52372edb444c317535d1d5a496c3e%40%3Cdev.flink.apache.org%3E > >> > > > >> > >> > >> > > > >> > >> Cheers, > >> > > > >> > >> Till > >> > > > >> > >> > >> > > > >> > > > >> > > > >> > > >> > > > >> > > >> > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > > >
