Thanks for the PR, Andy. I see that it has been merged, but I feel we haven't yet had the opportunity to discuss this in detail within the Iceberg community. Kevin and I were discussing this offline and feel that removing the broken code in its current state makes it difficult to restore in the future.
I propose the following: - Revert the removal PR. - Fix the broken code and include it in a patch release so we freeze a working version. - Discuss the removal during the next Iceberg community sync on April 1. - If we reach a consensus, re-apply the removal PR. What do you think? ~ Anurag On Wed, Mar 18, 2026 at 9:54 AM Andy Grove <[email protected]> wrote: > I have a draft PR up to remove the current Comet integration. I am > waiting for CI to finish before marking it as ready for review. > > https://github.com/apache/iceberg/pull/15674 > > On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <[email protected]> wrote: > >> Thanks for the update, Andy! >> It looks like most of the Comet code lives in the Spark integration [1]. >> We're currently working on the next 1.11 release. I know the removal isn't >> time-sensitive, but just wanted to mention it. >> >> Best, >> Kevin Liu >> >> [1] >> https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet >> >> On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <[email protected]> wrote: >> >>> Hi, >>> >>> The Comet community has made good progress with the iceberg-rust >>> integration and we now believe that this is the best path forward for >>> Comet, so are no longer going to be working on the integration with >>> Iceberg-Java. I posted a comment on the discussion issue as well [1]. >>> >>> Thanks for the community support with all of this. We will follow up >>> with a PR soon to remove the current (broken) Comet integration from >>> Iceberg-Java. >>> >>> Thanks, >>> >>> Andy. >>> >>> [1] >>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537 >>> >>> On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <[email protected]> >>> wrote: >>> >>>> Hi Team, >>>> The File Format API is merged, so that should not block the progress on >>>> the Comet integration progress anymore. >>>> If you need help, let me know. >>>> Thanks, >>>> Peter >>>> >>>> Andy Grove <[email protected]> ezt írta (időpont: 2026. jan. 22., >>>> Cs, 21:51): >>>> >>>>> I added another comment [1] on the issue [2], but will share here as >>>>> well for maximum visibility. >>>>> >>>>> I have two PRs in Comet that I am looking for reviews on. >>>>> >>>>> The first PR adds an @IcebergApi annotation to all Java >>>>> classes/methods/fields that are currently referenced from the Iceberg main >>>>> branch, and also adds documentation to the contributors guide. >>>>> #3237 <https://github.com/apache/datafusion-comet/pull/3237> >>>>> >>>>> The second PR builds on this and adds new dedicated unit tests for API >>>>> stability. This PR is more consequential because it discusses >>>>> deprecating/removing Comet's native_comet scan and how that relates >>>>> to the Iceberg API. >>>>> #3240 <https://github.com/apache/datafusion-comet/pull/3240> >>>>> >>>>> I am not an expert on the current integration, so it is possible that >>>>> I may have misunderstood things. I would appreciate reviews from both the >>>>> Iceberg and Comet communities! >>>>> Thanks, >>>>> >>>>> Andy. >>>>> >>>>> [1] >>>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915 >>>>> [2] https://github.com/apache/datafusion-comet/issues/2921 >>>>> >>>>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks for starting the thread! I've added a comment on the Github >>>>>> issue. Super exciting to see the iceberg-rust integration with Comet. >>>>>> As mentioned, I'll help take a look at >>>>>> https://github.com/apache/iceberg/pull/13786 and see if we can bring >>>>>> it to the finish line. >>>>>> >>>>>> Best, >>>>>> Kevin Liu >>>>>> >>>>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks Andy for raising this. >>>>>>> >>>>>>> +1 for the iceberg-rust solution. This could benefit the whole oss >>>>>>> community in the long term and avoid duplicated efforts. As with the >>>>>>> technique challenges you mentioned, this seems more like a missing >>>>>>> feature >>>>>>> rather than limitation. >>>>>>> >>>>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Andy for starting this discussion! >>>>>>>> >>>>>>>> I agree we should prioritize the iceberg-rust path as the long term >>>>>>>> default for Comet/Iceberg scans, while it's still valuable to keep the >>>>>>>> Iceberg Java integration as an option, especially for users who need >>>>>>>> JVM-specific features. I'm hoping we can get consensus on a path to >>>>>>>> resolve >>>>>>>> the shading issues (e.g. the approach proposed in >>>>>>>> github.com/apache/iceberg/pull/13786) so the java option can move >>>>>>>> forward. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Huaxin >>>>>>>> >>>>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Thanks Andy for starting this thread! >>>>>>>>> >>>>>>>>> I've replied to the issue above but am posting here as well for >>>>>>>>> better visibility: >>>>>>>>> >>>>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the >>>>>>>>> shading and circular dependency headaches of the Java approach. Easier >>>>>>>>> maintenance typically lowers the entry barrier and encourages more >>>>>>>>> community involvement, which is important for the project’s long-term >>>>>>>>> health. >>>>>>>>> >>>>>>>>> It also makes sense to keep the Iceberg Java integration available >>>>>>>>> as an experimental option for users who need its richer JVM-specific >>>>>>>>> features while Rust becomes the long-term default. We don’t want to >>>>>>>>> block >>>>>>>>> contributions from anyone who truly needs the Java path. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Shawn >>>>>>>>> >>>>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> There is a discussion in the DataFusion Comet community about the >>>>>>>>>> future direction of our Iceberg support [1]. For example, should we >>>>>>>>>> continue the efforts to integrate with the Iceberg Java project or >>>>>>>>>> focus >>>>>>>>>> more on the iceberg-rust project. >>>>>>>>>> >>>>>>>>>> It would be great to also get input from the Iceberg community. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Andy. >>>>>>>>>> >>>>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921 >>>>>>>>>> >>>>>>>>>
