> > Do you know if Hadoop/HBase is also using a maven plugin to fail a build on > breaking API changes? I would really like to have such a functionality in > Flink, because we can spot breaking changes very early.
I don't think we have maven integration for this as of yet. We release managers run a script $HBASE/dev-support/check_compatibility.sh that generates a source and binary compatibility report. Issues are then resolved during the period leading up to the release candidate. I think Hadoop is relying on a "QA bot" which reads patches from JIRA and > then does these > checks? > The "QA bot" is just a collection of shell scripts used during "Patch Available" status when a patch has been attached to JIRA or when a PR has been submitted through github. The check_compatibility script could be included in that automation, I don't see why not. Maybe you'd like to open a YETUS ticket :) I've pushed a branch to my own GitHub account with all classes I would make > public annotated: > > https://github.com/apache/flink/compare/master...rmetzger:interface_stability_revapi?expand=1 > Since this is really hard to read, I (half-automated) generated the > following list of annotated classes: > > https://github.com/rmetzger/flink/blob/interface_stability_revapi/annotations.md > > Please let me know if you would like to include or exclude classes from > that list. > Also, let me know which methods (in stable classes) you would mark as > experimental. > > > > > On Wed, Nov 11, 2015 at 10:17 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > > > +1 for some way of declaring public interfaces as experimental. > > > > > On 10 Nov 2015, at 22:24, Stephan Ewen <se...@apache.org> wrote: > > > > > > I think we need anyways an annotation "@PublicExperimental". > > > > > > We can make this annotation such that it can be added to methods and > can > > > use that to declare > > > Methods in an otherwise public class (such as DataSet) as experimental. > > > > > > On Tue, Nov 10, 2015 at 10:19 PM, Fabian Hueske <fhue...@gmail.com> > > wrote: > > > > > >> I am not sure if we always should declare complete classes as > > >> @PublicInterface. > > >> This does definitely make sense for interfaces and abstract classes > > such as > > >> MapFunction or InputFormat but not necessarily for classes such as > > DataSet > > >> that we might want to extend by methods which should not immediately > be > > >> considered as stable. > > >> > > >> > > >> 2015-11-10 21:36 GMT+01:00 Vasiliki Kalavri < > vasilikikala...@gmail.com > > >: > > >> > > >>> Yes, my opinion is that we shouldn't declare the Gelly API frozen > yet. > > >>> We can reconsider when we're closer to the 1.0 release, but if > > possible, > > >> I > > >>> would give it some more time. > > >>> > > >>> -V. > > >>> > > >>> On 10 November 2015 at 21:06, Stephan Ewen <se...@apache.org> wrote: > > >>> > > >>>> I think no component should be forced to be stable. It should be an > > >>>> individual decision for each component, and in some cases even for > > >>>> individual classes. > > >>>> > > >>>> @Vasia If you think Gelly should not be declared interface-frozen, > > then > > >>>> this is a good point to raise and this should definitely be > reflected. > > >>>> There is no point in declaring certain APIs as frozen when we are > not > > >> yet > > >>>> confident they have converged. > > >>>> > > >>>> > > >>>> > > >>>> On Tue, Nov 10, 2015 at 8:39 PM, Vasiliki Kalavri < > > >>>> vasilikikala...@gmail.com > > >>>>> wrote: > > >>>> > > >>>>> Hi Robert, > > >>>>> > > >>>>> thanks for bringing this up! > > >>>>> > > >>>>> I generally like the idea, but I wouldn't rush to annotate the > Gelly > > >>>>> classes yet. Gelly hasn't had that many users and I'm quite sure > > >> we'll > > >>>> find > > >>>>> things to improve as it gets more exposure. > > >>>>> TBH, I think it's quite unfair to force Gelly (also e.g. ML, Table) > > >> to > > >>> a > > >>>>> "1.0" status (from an API stability point of view) since they're > > >> really > > >>>>> young compared to the other Flink APIs. > > >>>>> > > >>>>> Cheers, > > >>>>> Vasia. > > >>>>> On Nov 10, 2015 8:04 PM, "Robert Metzger" <rmetz...@apache.org> > > >> wrote: > > >>>>> > > >>>>>> Hi everyone, > > >>>>>> > > >>>>>> I would like to bring this discussion back to your attention as we > > >>> seem > > >>>>> to > > >>>>>> approach the 1.0 release of Flink. > > >>>>>> > > >>>>>> My suggestion back in January was to annotate all classes, but I > > >>> think > > >>>>>> it'll be more feasible to just annotate public classes. > > >>>>>> So how about adding an annotation @PublicInterface > > >>>>>> > > >>>>>> For @PublicInterface, I would annotate classes such as: DataSet, > > >>>>>> DataStream, ExecutionEnvironment, InputFormat, MapFunction, > > >>> FileSystems > > >>>>> but > > >>>>>> also Gelly for example. > > >>>>>> > > >>>>>> I would not annotate as public components such as ML, Storm > > >>>>> compatibility, > > >>>>>> internals from runtime, yarn, optimizer. > > >>>>>> > > >>>>>> > > >>>>>> From a tooling perspective, I've looked into different maven > > >> plugins > > >>>> and > > >>>>>> java libraries and I found https://github.com/siom79/japicmp to > be > > >>> the > > >>>>>> closest to our needs. I actually opened a pull request to the > > >> project > > >>>> to > > >>>>>> allow inclusion/exclusion of classes based on annotations. Lets > > >> hope > > >>> it > > >>>>>> gets merged. > > >>>>>> > > >>>>>> Does everybody agree with adding just the @PublicInterface > > >>> annotation? > > >>>>>> > > >>>>>> Note that I'll add the annotation on a class-level, making the > > >> entire > > >>>>> class > > >>>>>> either public or private (from a stability point of view). If we > > >>> need a > > >>>>>> more fine-grained annotation, we have to add a second > > >>> @PrivateInterface > > >>>>>> annotation which we'll only apply to certain methods. > > >>>>>> > > >>>>>> The next step is that I'm going to open a pull request with all > > >>> classes > > >>>>>> annotated that I consider public. > > >>>>>> > > >>>>>> > > >>>>>> On Fri, Jan 30, 2015 at 7:10 PM, Henry Saputra < > > >>>> henry.sapu...@gmail.com> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> I like the idea. But would love to have different name for the > > >>>>>>> "LimitedPrivate" to make it easier to distinguish. > > >>>>>>> How about "Module" or "Package" ? > > >>>>>>> > > >>>>>>> - Henry > > >>>>>>> > > >>>>>>> On Wed, Jan 28, 2015 at 1:29 AM, Robert Metzger < > > >>> rmetz...@apache.org > > >>>>> > > >>>>>>> wrote: > > >>>>>>>> I think in Hadoop they use LimitedPrivate for the different > > >>>>> components > > >>>>>> of > > >>>>>>>> the project. > > >>>>>>>> For example LimitedPrivate("yarn"). > > >>>>>>>> Here is a very good documentation on the topic: > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html > > >>>>>>>> > > >>>>>>>> On Tue, Jan 27, 2015 at 3:58 PM, Alexander Alexandrov < > > >>>>>>>> alexander.s.alexand...@gmail.com> wrote: > > >>>>>>>> > > >>>>>>>>> I don't get the difference between Private and LimitedPrivate, > > >>> but > > >>>>>>>>> otherwise seems like quite a nice idea. > > >>>>>>>>> > > >>>>>>>>> It will be also good if we can agree upon what these tags > > >>> actually > > >>>>>> mean > > >>>>>>> and > > >>>>>>>>> add this meaning to the documentation. > > >>>>>>>>> > > >>>>>>>>> 2015-01-27 15:46 GMT+01:00 Robert Metzger < > > >> rmetz...@apache.org > > >>>> : > > >>>>>>>>> > > >>>>>>>>>> Hi, > > >>>>>>>>>> > > >>>>>>>>>> Hadoop has annotations for tagging the stability and > > >> audience > > >>> of > > >>>>>>> classes > > >>>>>>>>>> and methods. > > >>>>>>>>>> > > >>>>>>>>>> Through that, you can have @InterfaceAudience.Public, > > >> Private, > > >>>>>>>>>> LimitedPrivate > > >>>>>>>>>> and also @InterfaceStability.Evolving, Unstable, and Stable. > > >>>>>>>>>> > > >>>>>>>>>> I guess there are tools which allow to automatically check > > >> if > > >>>>>>> interfaces, > > >>>>>>>>>> which are marked as Stable have been changed between > > >> releases > > >>>> (or > > >>>>> by > > >>>>>>> pull > > >>>>>>>>>> requests). > > >>>>>>>>>> > > >>>>>>>>>> I think such annotations are crucial if we want to guarantee > > >>>>>> interface > > >>>>>>>>>> stability between releases. > > >>>>>>>>>> > > >>>>>>>>>> What do you think? Should we add those annotations? Which > > >> one > > >>>>> would > > >>>>>>> you > > >>>>>>>>>> like to add? > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Robert > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > >
