Hi Iceberg Community, It's been a while since we started this discussion. I'd like to revive the conversion for two reasons: 1) I think I'll have some capacity starting from early next year to take care of the C++ Puffin stuff we already talked about above, and also from the Impala community we could add some additional auxiliary functionality for the V3 positional deletes later on. 2) I learned that a part of the community is interested in having a C++ implementation of the Iceberg lib in general for their C++ engine. cc @Gang Wu
There seemed to be general support from the community to start up such a sub-project, so I'm reaching out now to ask for some guidance so that we can get going. @Fokko Driesprong <fo...@apache.org> You have much experience in this area, do you think you have the bandwidth to handhold us with the process and the steps? Meanwhile, I'll take a look at the Python repo to get a feel for myself. Regards, Gabor On Wed, Sep 11, 2024 at 5:32 PM Anton Okolnychyi <aokolnyc...@gmail.com> wrote: > If C++ engines prefer not to depend on Iceberg Rust, I actually don't see > a problem with having a separate C++ project even if it only contains > Puffin readers/writers. The important part is to avoid multiple C++ > writer/reader implementations in different engines. > > There were concerns with having one repo for multiple languages when we > discussed how to maintain the main project. We should re-evaluate if they > still apply here. > > - Anton > > вт, 10 вер. 2024 р. о 06:08 Gabor Kaszab <gaborkas...@apache.org> пише: > >> Hey All, >> >> Thanks for all the answers! >> >> My initial thought here was to have an iceberg-cpp repo for the Puffin >> reader/writer implementation in C++. But then I thought that there might be >> only one thing within this cpp repo, that is the Puffin stuff, and once >> implemented the repo won't have any more PRs coming in so it might be an >> overkill to introduce a separate repo for this purpose. Unless there is a >> need from other stakeholders to have a C++ implementation of the Iceberg >> lib (similarly to Python and Rust) but this would lead to a different >> conversation. >> >> Pulling in the Rust implementation into the C++ part of Impala: I don't >> know, I have to do some research on this. For performance reasons I know >> Impala prefers to have its own implementation of things (like Parquet >> reader/writer), so I have to double-check if it's acceptable >> performance-wise to pull in a Rust implementation of anything. I don't have >> any experience of doing so, hence the hesitation. >> >> JB, you mean that we can create a sub-repo like iceberg-puffin, that >> would hold separate implementations for different languages? For me this >> could also work. Wondering what others think. >> >> Regards, >> Gabor >> >> >> On Fri, Aug 30, 2024 at 2:08 PM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> Hi Gabor >>> >>> I like the idea, it sounds good to me. >>> >>> Imho, for "clarity", the best option would be to have a dedicated >>> puffin repo with different language binding (at bit like it's done in >>> Arrow). >>> I think that adding to iceberg-rust could be an option, but not sure >>> it would be obvious for developers. >>> >>> Regards >>> JB >>> >>> On Thu, Aug 29, 2024 at 2:46 PM Gabor Kaszab <gaborkas...@apache.org> >>> wrote: >>> > >>> > Hi Iceberg Community, >>> > >>> > With the V3 position delete proposal it came up that non-Java engines >>> might have to implement a Puffin reader and writer themselves so that they >>> can support the newly proposed position deletes. Impala would most probably >>> require a C++ implementation and first of all I'm wondering whether there >>> are other engines that require additional language implementation. >>> > >>> > My additional thought is that once we have an implementation in >>> another language where should that live? I can do the C++ and that could >>> live within Impala, but I think it could be useful for other engines too >>> but I have no idea where such an implementation could live TBH. >>> > >>> > Would be nice to hear opinions on this! >>> > Thanks, >>> > Gabor >>> >>
