Hi Konstantin and Francesco, > * Are you planning to implement the required tests via ArchUnit?
If ArchUnit is the best tool for the job, then I will probably use it. But I haven't investigated it yet. > * Fixing existing "test failures" is in the scope of this FLIP, correct? Yes I think that fixing the inconsistencies discovered by the to be added tests is part of the FLIP. > I think it would be nice that these docs are actually Javadocs, since our PublicEvolving javadocs tend to be already complete. perhaps there is a way we can play around the javadoc tool to generate only pages with @Public and @PublicEvolving on it. My plan was to state what the different @Public, @PublicEvolving, @Experimental annotations mean in the documentation. One can also find the definitions in the code, but I think some users don't look too closely at the code and JavaDocs. > TCKs I agree that TCKs/test suites, which ensure that a certain behaviour does not change, are important. However, I don't think that we have to define all TCKs in the context of this FLIP since it would let the scope explode and grind this effort to a halt. Instead, I would suggest that component shepherds start follow up discussions on how they can ensure that Flink's behaviour does not change for stable APIs and then add the corresponding TCKs/test suites. Cheers, Till On Fri, Dec 3, 2021 at 10:14 AM Francesco Guardiani <france...@ververica.com> wrote: > Hi Till, > > Thanks for starting this discussion, I think it's very beneficial for the > community to have stable APIs, in particular to develop connectors and > formats. > > A couple of comments: > > > I would suggest that we document these guarantees prominently under > /docs/dev/api_stability. > > I think it would be nice that these docs are actually Javadocs, since our > PublicEvolving javadocs tend to be already complete. perhaps there is a way > we can play around the javadoc tool to generate only pages with @Public and > @PublicEvolving on it. > > > Test Plan > > IMHO this proposal lacks what testing means from the user perspective: I > think a discussion about API stability should go hand to hand with a > discussion about providing TCKs to users. If the interfaces to build a > Source are claimed to be stable, then, from the flink runtime point of > view, a 3rd party source implementation is expected to have certain > behaviors. On the other hand, as a 3rd party developer, I expect flink > gives me some means to test if my source is compliant with the basic > "expected behaviors". This is even more relevant when you're building a > source for Table, where we have well defined behaviors through our > "abilities" interfaces. > > TCKs also have another very important aspect: they serve as a very detailed > documentation of all the behaviors that comes in/out a specific interface. > > Another thing to mention is that we already have some of these test, they > just need to be polished to make them publicly available; e.g. look at the > JsonBatchFileSystemITCase, it uses all the test cases from > FileSystemITCaseBase to test that the format works correctly with the > filesystem source. > > A TCK is something that can take quite some time to be considered > "complete", so I'm not suggesting at all to get it done all at once inside > this FLIP. But I think that if we bootstrap it in this FLIP, we can then > progressively add new test cases every time we should add them internally, > or when we need to rework the existing ones. > > > FG > > On Fri, Dec 3, 2021 at 9:32 AM Konstantin Knauf <kna...@apache.org> wrote: > > > Hi Till, > > > > Thank you for picking up this topic. Explicit and consistent stability > > guarantees are important for our users as well as any downstream project > of > > Apache Flink. Documenting the de-facto status quo and tackling existing > > inconsistencies sounds like a good first step. So, +1 from my side. > > > > Two questions for clarity: > > * Are you planning to implement the required tests via ArchUnit? > > * Fixing existing "test failures" is in the scope of this FLIP, correct? > > > > Cheers, > > > > Konstantin > > > > On Thu, Dec 2, 2021 at 3:48 PM Till Rohrmann <trohrm...@apache.org> > wrote: > > > > > Hi everyone, > > > > > > I would like to start a discussion about what kind of API stability > > > guarantees we want to provide to our users. The Flink community already > > > agreed on some stability guarantees, but these guarantees were only > > > communicated via the ML and not properly documented [2]. Moreover, I've > > > seen more and more complaints about breaking changes (some rightful and > > > others not) on the ML recently [3] because we rarely mark our APIs as > > > stable. This motivated this FLIP. > > > > > > The proposal first concentrates on source API stability guarantees. > > Binary > > > stability guarantees are explicitly left out for a follow-up > discussion. > > > > > > In essence, the proposal follows our current stability guarantees while > > > updating the guarantees for @PublicEvolving that we are already > providing > > > w/o having stated them. Moreover, this FLIP proposes some guidelines > for > > > determining the stability guarantees for an API object, how to evolve > > them > > > and some additional requirements for the return and parameter types of > > > methods. > > > > > > All in all, I hope that this proposal is more reflecting our current > > > understanding of stability guarantees than being controversial. One of > > the > > > outcomes of this FLIP should be to properly document these guarantees > so > > > that it is easily discoverable and understandable for our users. > > Moreover, > > > I hope that we can provide more stable APIs our users can rely and > build > > > upon. > > > > > > There will be a follow-up FLIP discussing the problem of how to make > sure > > > that APIs become stable over time. > > > > > > Looking forward to your feedback. > > > > > > [1] https://cwiki.apache.org/confluence/x/IJeqCw > > > [2] https://lists.apache.org/thread/5jm25783oq5svyk7rr8g1gly2ooxqhjr > > > [3] https://lists.apache.org/thread/kzhfc3t6omzo2kyo8zj9yxoh8twq5fzr > > > > > > Cheers, > > > Till > > > > > > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > > >
