Fyi: Ingo already implemented most of the FLIP by introducing the ApiAnnotationRules.PUBLIC_API_METHODS_USE_ONLY_PUBLIC_API_TYPES and .PUBLIC_EVOLVING_API_METHODS_USE_ONLY_PUBLIC_EVOLVING_API_TYPES [1] using ArchUnit :-)
[1] https://github.com/apache/flink/blob/master/flink-architecture-tests/src/test/java/org/apache/flink/architecture/rules/ApiAnnotationRules.java Cheers, Till On Fri, Dec 3, 2021 at 11:59 AM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Konstantin and Francesco, > > > * Are you planning to implement the required tests via ArchUnit? > > If ArchUnit is the best tool for the job, then I will probably use it. But > I haven't investigated it yet. > > > * Fixing existing "test failures" is in the scope of this FLIP, correct? > > Yes I think that fixing the inconsistencies discovered by the to be added > tests is part of the FLIP. > > > I think it would be nice that these docs are actually Javadocs, since > our PublicEvolving javadocs tend to be already complete. perhaps there is a > way we can play around the javadoc tool to generate only pages with @Public > and @PublicEvolving on it. > > My plan was to state what the > different @Public, @PublicEvolving, @Experimental annotations mean in the > documentation. One can also find the definitions in the code, but I think > some users don't look too closely at the code and JavaDocs. > > > TCKs > > I agree that TCKs/test suites, which ensure that a certain behaviour does > not change, are important. However, I don't think that we have to define > all TCKs in the context of this FLIP since it would let the scope explode > and grind this effort to a halt. Instead, I would suggest that component > shepherds start follow up discussions on how they can ensure that Flink's > behaviour does not change for stable APIs and then add the corresponding > TCKs/test suites. > > Cheers, > Till > > On Fri, Dec 3, 2021 at 10:14 AM Francesco Guardiani < > france...@ververica.com> wrote: > >> Hi Till, >> >> Thanks for starting this discussion, I think it's very beneficial for the >> community to have stable APIs, in particular to develop connectors and >> formats. >> >> A couple of comments: >> >> > I would suggest that we document these guarantees prominently under >> /docs/dev/api_stability. >> >> I think it would be nice that these docs are actually Javadocs, since our >> PublicEvolving javadocs tend to be already complete. perhaps there is a >> way >> we can play around the javadoc tool to generate only pages with @Public >> and >> @PublicEvolving on it. >> >> > Test Plan >> >> IMHO this proposal lacks what testing means from the user perspective: I >> think a discussion about API stability should go hand to hand with a >> discussion about providing TCKs to users. If the interfaces to build a >> Source are claimed to be stable, then, from the flink runtime point of >> view, a 3rd party source implementation is expected to have certain >> behaviors. On the other hand, as a 3rd party developer, I expect flink >> gives me some means to test if my source is compliant with the basic >> "expected behaviors". This is even more relevant when you're building a >> source for Table, where we have well defined behaviors through our >> "abilities" interfaces. >> >> TCKs also have another very important aspect: they serve as a very >> detailed >> documentation of all the behaviors that comes in/out a specific interface. >> >> Another thing to mention is that we already have some of these test, they >> just need to be polished to make them publicly available; e.g. look at the >> JsonBatchFileSystemITCase, it uses all the test cases from >> FileSystemITCaseBase to test that the format works correctly with the >> filesystem source. >> >> A TCK is something that can take quite some time to be considered >> "complete", so I'm not suggesting at all to get it done all at once inside >> this FLIP. But I think that if we bootstrap it in this FLIP, we can then >> progressively add new test cases every time we should add them internally, >> or when we need to rework the existing ones. >> >> >> FG >> >> On Fri, Dec 3, 2021 at 9:32 AM Konstantin Knauf <kna...@apache.org> >> wrote: >> >> > Hi Till, >> > >> > Thank you for picking up this topic. Explicit and consistent stability >> > guarantees are important for our users as well as any downstream >> project of >> > Apache Flink. Documenting the de-facto status quo and tackling existing >> > inconsistencies sounds like a good first step. So, +1 from my side. >> > >> > Two questions for clarity: >> > * Are you planning to implement the required tests via ArchUnit? >> > * Fixing existing "test failures" is in the scope of this FLIP, correct? >> > >> > Cheers, >> > >> > Konstantin >> > >> > On Thu, Dec 2, 2021 at 3:48 PM Till Rohrmann <trohrm...@apache.org> >> wrote: >> > >> > > Hi everyone, >> > > >> > > I would like to start a discussion about what kind of API stability >> > > guarantees we want to provide to our users. The Flink community >> already >> > > agreed on some stability guarantees, but these guarantees were only >> > > communicated via the ML and not properly documented [2]. Moreover, >> I've >> > > seen more and more complaints about breaking changes (some rightful >> and >> > > others not) on the ML recently [3] because we rarely mark our APIs as >> > > stable. This motivated this FLIP. >> > > >> > > The proposal first concentrates on source API stability guarantees. >> > Binary >> > > stability guarantees are explicitly left out for a follow-up >> discussion. >> > > >> > > In essence, the proposal follows our current stability guarantees >> while >> > > updating the guarantees for @PublicEvolving that we are already >> providing >> > > w/o having stated them. Moreover, this FLIP proposes some guidelines >> for >> > > determining the stability guarantees for an API object, how to evolve >> > them >> > > and some additional requirements for the return and parameter types of >> > > methods. >> > > >> > > All in all, I hope that this proposal is more reflecting our current >> > > understanding of stability guarantees than being controversial. One of >> > the >> > > outcomes of this FLIP should be to properly document these guarantees >> so >> > > that it is easily discoverable and understandable for our users. >> > Moreover, >> > > I hope that we can provide more stable APIs our users can rely and >> build >> > > upon. >> > > >> > > There will be a follow-up FLIP discussing the problem of how to make >> sure >> > > that APIs become stable over time. >> > > >> > > Looking forward to your feedback. >> > > >> > > [1] https://cwiki.apache.org/confluence/x/IJeqCw >> > > [2] https://lists.apache.org/thread/5jm25783oq5svyk7rr8g1gly2ooxqhjr >> > > [3] https://lists.apache.org/thread/kzhfc3t6omzo2kyo8zj9yxoh8twq5fzr >> > > >> > > Cheers, >> > > Till >> > > >> > >> > >> > -- >> > >> > Konstantin Knauf >> > >> > https://twitter.com/snntrable >> > >> > https://github.com/knaufk >> > >> >
