Thanks for the information, Martijin & Timo ! > Since implementing a connector is not straightforward, we were expecting that not many users implement custom connectors.
Currently, the apache iceberg & hudi are heavily depending on the PublicEvolving API for their flink connectors. I think apache hudi even uses more public API than iceberg to implement their relatively complicated flink sink DAG, I think Danny Chen [1] may want to provide more input. API compatibility has become one of the core reasons that downstream projects maintainers vote to support a release or not because bandwidth from the downstream projects are limited and we maintainers need to balance between the community requirements and cost. A great compatible flink release will greatly save the maintenance cost (especially we flink release often ) and we are also glad to make it a longer life cycle. > We therefore consider this part as a kind of "second level API" for which we can evolve quicker. That sounds great ! I'm glad to see that we are making the API more friendly ! [1]. https://github.com/danny0405 On Tue, Sep 28, 2021 at 3:52 PM Timo Walther <twal...@apache.org> wrote: > Hi Zheng, > > I'm very sorry for the inconvenience that we have caused with our API > changes. We are trying our best to avoid API breaking changes. Thanks > for giving us feedback. > > There has been a reason why Table API was marked as @PublicEvolving > instead of @Public. Over the last two years, we have basically rewritten > the entire API [1] to digest the Blink merge and making the Table API > stable and ready for the future. We tried our best to give users 1-2 > releases time to upgrade their implementations whenever we deprecated > API but we were aware that this might cause frustration, but hopefully > for the greater good. We have reworked type system, Catalog API, schema, > source/sinks, functions and much more. Flink 1.14 will hopefully be the > last release with major API changes. We could also mark most Table API > interfaces as `@Public` in 1.15. > > For your mentioned incompatibility, I agree that the change from > CatalogTable to ResolvedCatalogTable was not very nice. Since > implementing a connector is not straight forward, we were expecting that > not many users implement custom connectors. We therefore consider this > part as kind of "second level API" for which we can evolve quicker. A > `context.getCatalogTable().getSchema()` should still work for 1.12 and > 1.13, at least that was the goal. > > Thanks again for the feedback. It was a good reminder and we will pay > more attention to this. > > Regards, > Timo > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions > > > On 28.09.21 08:40, Martijn Visser wrote: > > Hi Zheng, > > > > Thanks for reaching out and sharing your frustration. No feelings are > hurt > > and feedback is always welcome, because that's the only way we can > improve > > for the future. API compatibility is a really important thing for us > while > > also improving and building new capabilities. Let me investigate a bit > what > > happened on our end, share that and then try to get some learnings out of > > it for the future. I'll get back to you in a couple of days. > > > > Best regards, > > > > Martijn Visser | Product Manager > > > > mart...@ververica.com > > > > <https://www.ververica.com/> > > > > > > Follow us @VervericaData > > > > -- > > > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > > Conference > > > > Stream Processing | Event Driven | Real Time > > > > > > On Tue, 28 Sept 2021 at 07:39, OpenInx <open...@gmail.com> wrote: > > > >> Sorry about my unfriendly tone of the last e-mail, I got frustrated > about > >> the experience of maintaining the project which is closely with Flink. > My > >> intention was trying to remind everyone to be careful about API > >> compatibility and didn't really watch out for the tone I used. > >> > >> Hope that doesn't hurt anyone's feelings. > >> > >> On Tue, Sep 28, 2021 at 12:33 PM OpenInx <open...@gmail.com> wrote: > >> > >>> Hi Dev > >>> > >>> We are trying to upgrade the flink version from 1.12.0 to 1.13.2 in > >> apache > >>> iceberg project ( https://github.com/apache/iceberg/pull/3116), but > >> it's > >>> not a great experience. We expect to support both flink1.12 and > >> flink1.13 > >>> in an iceberg-flink module without using the new API of flink1.13 for > >>> saving maintenance cost, but we find the iceberg-flink-runtime.jar > built > >>> by flink 1.13 cannot works fine in flink 1.12 clusters because of the > >> basic > >>> API compatibility was break when iterating flink 1.12 to flink1.13.2: > >>> > >>> (The following are copied from the iceberg issue: > >>> https://github.com/apache/iceberg/issues/3187#issuecomment-928755046) > >>> > >>> Thanks for the report, @Reo-LEI ! I think this issue was introduced > from > >>> this apache flink PR ( > >>> > >> > https://github.com/apache/flink/pull/15316/files#diff-bd276ed951054125b39428ee61de103d9c7832246398f01514a574bb8e51757cR74 > >> ) > >>> and FLINK-21913 (https://issues.apache.org/jira/browse/FLINK-21913), > it > >>> just changed the returned data type from CatalogTable to > >>> ResolvedCatalogTable without any compatibility guarantee. In this case, > >> the > >>> iceberg-flink-runtime jar which is compiled from apache flink 1.13 will > >>> include the ResovledCatalogTable class inside it. Finally when we > package > >>> this jar and submit the flink job to flink 1.12, the above > compatibility > >>> issue happen. > >>> > >>> As we all know, the DynamicTableFactory ( > >>> > >> > https://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java > >> ) > >>> is a basic API which almost all flink connectors are built on top of > it. > >>> The breaking compatibility makes the downstream projects really hard to > >>> deliver better compatibility to users, unless we iceberg maintain > >> different > >>> modules for each maintained flink version (That's not the thing that we > >>> want to do). > >>> > >>> The last flink upgrading work is also not a good experience (See the > >>> discussion (https://github.com/apache/iceberg/pull/1956) and comment ( > >>> https://github.com/apache/iceberg/pull/1956#discussion_r546534299) ), > >>> because the flink 1.12 also breaks several API that was annotated > >>> PublicEvolving in flink 1.11.0, that becomes one of the most important > >>> reasons leading to the conclusion that stops support flink 1.11.0 in > our > >>> apache iceberg branch ( Supporting new features [such as flip-27 > unified > >>> iceberg source/sink] that depends the API introduced in flink 1.12 is > >>> another reason). To better support the compatibility of downstream > >> systems > >>> and delivering better experience to flink users, I will strongly > suggest > >>> the Apache Flink community to pay more attention to ensuring API > >>> compatibility. > >>> > >>> > >>> Zheng Hu (openinx) > >>> > >>> Thanks. > >>> > >> > > > >
