Hi JB, Yes, Flink 2.0 support ideally should land in Iceberg 1.9.
Hi Rodrigo, +1 on closing the gap between the two sink implementations: FlinkSink and IcebergSink. Thanks for helping to close that gap! Hi Steven, Good point on the "Remove by Iceberg 2.0" claim stated in FlinkSource. We can keep it until Iceberg 2.0 in the Flink 1.X paths, but I would propose to remove FlinkSource earlier than Iceberg 2.0 for the Flink 2.X path. The reason is that it is not used by default for many Iceberg versions. I think it is unlikely that users jumping on Flink 2.0 will want to use it. As for removing FlinkSink, you're right that IcebergSink is relatively new as a replacement. If we can keep it in the process of porting the code, we may keep it around longer, but I suggest deprecating it and scheduling it for removal in the Iceberg release following 1.9. To summarize, I'm proposing the following: Iceberg 1.9 - Remove FlinkSource in Flink 2.0 code path - Deprecate FlinkSink for all supported Flink versions Iceberg > 1.9 - Remove FlinkSource for all supported Flink versions - Remove FlinkSink for all supported Flink versions Does that sound right? Cheers, Max On Thu, Mar 6, 2025 at 9:10 PM Steven Wu <stevenz...@gmail.com> wrote: > > > if Flink 2.0 release is going to release the old source and sink > > interfaces, t > > typo above: "release the old" -> "delete the old" > > On Thu, Mar 6, 2025 at 12:08 PM Steven Wu <stevenz...@gmail.com> wrote: >> >> There was a previous thread for the Flink source. The Javadoc deprecation >> note for the old `FlinkSource` currently says it will be removed in Iceberg >> 2.0 release >> https://lists.apache.org/thread/27kcvo3p86pysk9wrggq4vphzo03sv3l >> >> Now regarding deprecating and removing the old `FlinkSink` in favor of the >> new `IcebergSink`, the new `IcebergSink` was added 6 months ago and released >> in Iceberg 1.7 (current release is 1.8). Not sure how many users have got a >> chance to use it. Ideally, I would like to have the new `IcebergSink` bake >> longer with more users running it in the production environments. There is >> also the parity problems that Rod mentioned. >> >> However, if Flink 2.0 release is going to release the old source and sink >> interfaces, then we will have no choice and remove the old source and sink >> implementations earlier than we originally planned/preferred. >> >> >> On Thu, Mar 6, 2025 at 8:40 AM Rodrigo Meneses <rmene...@gmail.com> wrote: >>> >>> Hi Max, +1 on that too. IcebergSink has been hanging around for a while >>> now. We want to make sure we have the same features, though: >>> >>> I’ve got https://github.com/apache/iceberg/pull/12071 that adds the Range >>> Distribution Mode to IcebergSink. It still needs a few recent bug fixes and >>> features backported, but it should be ready soon. >>> >>> >>> >>> On Thu, Mar 6, 2025 at 8:18 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>>> >>>> Hi Max, >>>> >>>> I guess you are proposing to remove FlinkSink and the corresponding >>>> FlinkSource as well. >>>> >>>> It makes to me as, I saw in the code, that both FlinkSink and >>>> FlinkSource are deprecated for a while. >>>> >>>> So, +1 to remove it. >>>> >>>> Are you planning this for 1.9.0 ? >>>> >>>> Regards >>>> JB >>>> >>>> On Thu, Mar 6, 2025 at 4:21 PM Maximilian Michels <m...@apache.org> wrote: >>>> > >>>> > Hi, >>>> > >>>> > Today there are two Flink write connectors in Iceberg: >>>> > >>>> > 1. FlinkSink (original sink, based on Flink legacy interfaces) >>>> > 2. IcebergSink (newer version, based on modern Flink API) >>>> > >>>> > In terms of features, (1) is a subset of (2). >>>> > >>>> > I'm in the process of adding support for Flink 2.0. The interfaces >>>> > used for (1) have been deprecated for several Flink versions and are >>>> > removed / discouraged in Flink 2.0. >>>> > >>>> > Therefore, I would like to propose to remove FlinkSink for the Flink >>>> > 2.0 Iceberg module. We have already deprecated FlinkSink for a while. >>>> > >>>> > Any objections? >>>> > >>>> > -Max
