Hi Max Yes, it makes sense. Maybe I can help on that as I'm working on Spec V3 support in Flink (I did the default value, and I'm checking the new types now).
Please let me know :) Thanks, Regards JB On Tue, Mar 11, 2025 at 11:23 AM Maximilian Michels <m...@apache.org> wrote: > > Hi JB, > > Yes, Flink 2.0 support ideally should land in Iceberg 1.9. > > Hi Rodrigo, > > +1 on closing the gap between the two sink implementations: FlinkSink > and IcebergSink. Thanks for helping to close that gap! > > Hi Steven, > > Good point on the "Remove by Iceberg 2.0" claim stated in FlinkSource. > We can keep it until Iceberg 2.0 in the Flink 1.X paths, but I would > propose to remove FlinkSource earlier than Iceberg 2.0 for the Flink > 2.X path. The reason is that it is not used by default for many > Iceberg versions. I think it is unlikely that users jumping on Flink > 2.0 will want to use it. > > As for removing FlinkSink, you're right that IcebergSink is relatively > new as a replacement. If we can keep it in the process of porting the > code, we may keep it around longer, but I suggest deprecating it and > scheduling it for removal in the Iceberg release following 1.9. > > To summarize, I'm proposing the following: > > Iceberg 1.9 > - Remove FlinkSource in Flink 2.0 code path > - Deprecate FlinkSink for all supported Flink versions > > Iceberg > 1.9 > - Remove FlinkSource for all supported Flink versions > - Remove FlinkSink for all supported Flink versions > > Does that sound right? > > Cheers, > Max > > On Thu, Mar 6, 2025 at 9:10 PM Steven Wu <stevenz...@gmail.com> wrote: > > > > > if Flink 2.0 release is going to release the old source and sink > > > interfaces, t > > > > typo above: "release the old" -> "delete the old" > > > > On Thu, Mar 6, 2025 at 12:08 PM Steven Wu <stevenz...@gmail.com> wrote: > >> > >> There was a previous thread for the Flink source. The Javadoc deprecation > >> note for the old `FlinkSource` currently says it will be removed in > >> Iceberg 2.0 release > >> https://lists.apache.org/thread/27kcvo3p86pysk9wrggq4vphzo03sv3l > >> > >> Now regarding deprecating and removing the old `FlinkSink` in favor of the > >> new `IcebergSink`, the new `IcebergSink` was added 6 months ago and > >> released in Iceberg 1.7 (current release is 1.8). Not sure how many users > >> have got a chance to use it. Ideally, I would like to have the new > >> `IcebergSink` bake longer with more users running it in the production > >> environments. There is also the parity problems that Rod mentioned. > >> > >> However, if Flink 2.0 release is going to release the old source and sink > >> interfaces, then we will have no choice and remove the old source and sink > >> implementations earlier than we originally planned/preferred. > >> > >> > >> On Thu, Mar 6, 2025 at 8:40 AM Rodrigo Meneses <rmene...@gmail.com> wrote: > >>> > >>> Hi Max, +1 on that too. IcebergSink has been hanging around for a while > >>> now. We want to make sure we have the same features, though: > >>> > >>> I’ve got https://github.com/apache/iceberg/pull/12071 that adds the Range > >>> Distribution Mode to IcebergSink. It still needs a few recent bug fixes > >>> and features backported, but it should be ready soon. > >>> > >>> > >>> > >>> On Thu, Mar 6, 2025 at 8:18 AM Jean-Baptiste Onofré <j...@nanthrax.net> > >>> wrote: > >>>> > >>>> Hi Max, > >>>> > >>>> I guess you are proposing to remove FlinkSink and the corresponding > >>>> FlinkSource as well. > >>>> > >>>> It makes to me as, I saw in the code, that both FlinkSink and > >>>> FlinkSource are deprecated for a while. > >>>> > >>>> So, +1 to remove it. > >>>> > >>>> Are you planning this for 1.9.0 ? > >>>> > >>>> Regards > >>>> JB > >>>> > >>>> On Thu, Mar 6, 2025 at 4:21 PM Maximilian Michels <m...@apache.org> > >>>> wrote: > >>>> > > >>>> > Hi, > >>>> > > >>>> > Today there are two Flink write connectors in Iceberg: > >>>> > > >>>> > 1. FlinkSink (original sink, based on Flink legacy interfaces) > >>>> > 2. IcebergSink (newer version, based on modern Flink API) > >>>> > > >>>> > In terms of features, (1) is a subset of (2). > >>>> > > >>>> > I'm in the process of adding support for Flink 2.0. The interfaces > >>>> > used for (1) have been deprecated for several Flink versions and are > >>>> > removed / discouraged in Flink 2.0. > >>>> > > >>>> > Therefore, I would like to propose to remove FlinkSink for the Flink > >>>> > 2.0 Iceberg module. We have already deprecated FlinkSink for a while. > >>>> > > >>>> > Any objections? > >>>> > > >>>> > -Max
