I have a draft PR up to remove the current Comet integration. I am waiting for CI to finish before marking it as ready for review.
https://github.com/apache/iceberg/pull/15674 On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <[email protected]> wrote: > Thanks for the update, Andy! > It looks like most of the Comet code lives in the Spark integration [1]. > We're currently working on the next 1.11 release. I know the removal isn't > time-sensitive, but just wanted to mention it. > > Best, > Kevin Liu > > [1] > https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet > > On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <[email protected]> wrote: > >> Hi, >> >> The Comet community has made good progress with the iceberg-rust >> integration and we now believe that this is the best path forward for >> Comet, so are no longer going to be working on the integration with >> Iceberg-Java. I posted a comment on the discussion issue as well [1]. >> >> Thanks for the community support with all of this. We will follow up with >> a PR soon to remove the current (broken) Comet integration from >> Iceberg-Java. >> >> Thanks, >> >> Andy. >> >> [1] >> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537 >> >> On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <[email protected]> >> wrote: >> >>> Hi Team, >>> The File Format API is merged, so that should not block the progress on >>> the Comet integration progress anymore. >>> If you need help, let me know. >>> Thanks, >>> Peter >>> >>> Andy Grove <[email protected]> ezt írta (időpont: 2026. jan. 22., >>> Cs, 21:51): >>> >>>> I added another comment [1] on the issue [2], but will share here as >>>> well for maximum visibility. >>>> >>>> I have two PRs in Comet that I am looking for reviews on. >>>> >>>> The first PR adds an @IcebergApi annotation to all Java >>>> classes/methods/fields that are currently referenced from the Iceberg main >>>> branch, and also adds documentation to the contributors guide. >>>> #3237 <https://github.com/apache/datafusion-comet/pull/3237> >>>> >>>> The second PR builds on this and adds new dedicated unit tests for API >>>> stability. This PR is more consequential because it discusses >>>> deprecating/removing Comet's native_comet scan and how that relates to >>>> the Iceberg API. >>>> #3240 <https://github.com/apache/datafusion-comet/pull/3240> >>>> >>>> I am not an expert on the current integration, so it is possible that I >>>> may have misunderstood things. I would appreciate reviews from both the >>>> Iceberg and Comet communities! >>>> Thanks, >>>> >>>> Andy. >>>> >>>> [1] >>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915 >>>> [2] https://github.com/apache/datafusion-comet/issues/2921 >>>> >>>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <[email protected]> >>>> wrote: >>>> >>>>> Thanks for starting the thread! I've added a comment on the Github >>>>> issue. Super exciting to see the iceberg-rust integration with Comet. >>>>> As mentioned, I'll help take a look at >>>>> https://github.com/apache/iceberg/pull/13786 and see if we can bring >>>>> it to the finish line. >>>>> >>>>> Best, >>>>> Kevin Liu >>>>> >>>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks Andy for raising this. >>>>>> >>>>>> +1 for the iceberg-rust solution. This could benefit the whole oss >>>>>> community in the long term and avoid duplicated efforts. As with the >>>>>> technique challenges you mentioned, this seems more like a missing >>>>>> feature >>>>>> rather than limitation. >>>>>> >>>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks Andy for starting this discussion! >>>>>>> >>>>>>> I agree we should prioritize the iceberg-rust path as the long term >>>>>>> default for Comet/Iceberg scans, while it's still valuable to keep the >>>>>>> Iceberg Java integration as an option, especially for users who need >>>>>>> JVM-specific features. I'm hoping we can get consensus on a path to >>>>>>> resolve >>>>>>> the shading issues (e.g. the approach proposed in >>>>>>> github.com/apache/iceberg/pull/13786) so the java option can move >>>>>>> forward. >>>>>>> >>>>>>> Thanks, >>>>>>> Huaxin >>>>>>> >>>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Andy for starting this thread! >>>>>>>> >>>>>>>> I've replied to the issue above but am posting here as well for >>>>>>>> better visibility: >>>>>>>> >>>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the >>>>>>>> shading and circular dependency headaches of the Java approach. Easier >>>>>>>> maintenance typically lowers the entry barrier and encourages more >>>>>>>> community involvement, which is important for the project’s long-term >>>>>>>> health. >>>>>>>> >>>>>>>> It also makes sense to keep the Iceberg Java integration available >>>>>>>> as an experimental option for users who need its richer JVM-specific >>>>>>>> features while Rust becomes the long-term default. We don’t want to >>>>>>>> block >>>>>>>> contributions from anyone who truly needs the Java path. >>>>>>>> >>>>>>>> Best, >>>>>>>> Shawn >>>>>>>> >>>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> There is a discussion in the DataFusion Comet community about the >>>>>>>>> future direction of our Iceberg support [1]. For example, should we >>>>>>>>> continue the efforts to integrate with the Iceberg Java project or >>>>>>>>> focus >>>>>>>>> more on the iceberg-rust project. >>>>>>>>> >>>>>>>>> It would be great to also get input from the Iceberg community. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Andy. >>>>>>>>> >>>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921 >>>>>>>>> >>>>>>>>
