Thanks Gabor for the update! Yes, I think the iceberg-rust and iceberg-go are going on pretty well. I have an interest in working on a brand new iceberg-cpp project.
I have done some research and found that this is a long-awaited project but has made no progress so far. I'd like to hear the voice from the community to see if it is something that is worth doing. Best, Gang On Fri, Nov 22, 2024 at 4:26 PM Gabor Kaszab <gaborkas...@apache.org> wrote: > Hi Iceberg Community, > > It's been a while since we started this discussion. I'd like to revive the > conversion for two reasons: > 1) I think I'll have some capacity starting from early next year to take > care of the C++ Puffin stuff we already talked about above, and also from > the Impala community we could add some additional auxiliary functionality > for the V3 positional deletes later on. > 2) I learned that a part of the community is interested in having a C++ > implementation of the Iceberg lib in general for their C++ engine. cc @Gang > Wu > > There seemed to be general support from the community to start up such a > sub-project, so I'm reaching out now to ask for some guidance so that we > can get going. @Fokko Driesprong <fo...@apache.org> You have much > experience in this area, do you think you have the bandwidth to handhold us > with the process and the steps? Meanwhile, I'll take a look at the Python > repo to get a feel for myself. > > Regards, > Gabor > > On Wed, Sep 11, 2024 at 5:32 PM Anton Okolnychyi <aokolnyc...@gmail.com> > wrote: > >> If C++ engines prefer not to depend on Iceberg Rust, I actually don't see >> a problem with having a separate C++ project even if it only contains >> Puffin readers/writers. The important part is to avoid multiple C++ >> writer/reader implementations in different engines. >> >> There were concerns with having one repo for multiple languages when we >> discussed how to maintain the main project. We should re-evaluate if they >> still apply here. >> >> - Anton >> >> вт, 10 вер. 2024 р. о 06:08 Gabor Kaszab <gaborkas...@apache.org> пише: >> >>> Hey All, >>> >>> Thanks for all the answers! >>> >>> My initial thought here was to have an iceberg-cpp repo for the Puffin >>> reader/writer implementation in C++. But then I thought that there might be >>> only one thing within this cpp repo, that is the Puffin stuff, and once >>> implemented the repo won't have any more PRs coming in so it might be an >>> overkill to introduce a separate repo for this purpose. Unless there is a >>> need from other stakeholders to have a C++ implementation of the Iceberg >>> lib (similarly to Python and Rust) but this would lead to a different >>> conversation. >>> >>> Pulling in the Rust implementation into the C++ part of Impala: I don't >>> know, I have to do some research on this. For performance reasons I know >>> Impala prefers to have its own implementation of things (like Parquet >>> reader/writer), so I have to double-check if it's acceptable >>> performance-wise to pull in a Rust implementation of anything. I don't have >>> any experience of doing so, hence the hesitation. >>> >>> JB, you mean that we can create a sub-repo like iceberg-puffin, that >>> would hold separate implementations for different languages? For me this >>> could also work. Wondering what others think. >>> >>> Regards, >>> Gabor >>> >>> >>> On Fri, Aug 30, 2024 at 2:08 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>>> Hi Gabor >>>> >>>> I like the idea, it sounds good to me. >>>> >>>> Imho, for "clarity", the best option would be to have a dedicated >>>> puffin repo with different language binding (at bit like it's done in >>>> Arrow). >>>> I think that adding to iceberg-rust could be an option, but not sure >>>> it would be obvious for developers. >>>> >>>> Regards >>>> JB >>>> >>>> On Thu, Aug 29, 2024 at 2:46 PM Gabor Kaszab <gaborkas...@apache.org> >>>> wrote: >>>> > >>>> > Hi Iceberg Community, >>>> > >>>> > With the V3 position delete proposal it came up that non-Java engines >>>> might have to implement a Puffin reader and writer themselves so that they >>>> can support the newly proposed position deletes. Impala would most probably >>>> require a C++ implementation and first of all I'm wondering whether there >>>> are other engines that require additional language implementation. >>>> > >>>> > My additional thought is that once we have an implementation in >>>> another language where should that live? I can do the C++ and that could >>>> live within Impala, but I think it could be useful for other engines too >>>> but I have no idea where such an implementation could live TBH. >>>> > >>>> > Would be nice to hear opinions on this! >>>> > Thanks, >>>> > Gabor >>>> >>>
