Thanks for the support, Fokko and JB! Please include me in the cpp slack channel for future cooperation.
Best, Gang On Fri, Nov 22, 2024 at 4:58 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Gabor, > > I think it makes sense to create iceberg-cpp resources (repository, > slack channel, ...): this can gather the efforts in the Iceberg lib > and Puffin implementation. > > Fokko can help there, he can ping me if needed (from ASF standpoint) :) > > Regards > JB > > On Fri, Nov 22, 2024 at 9:25 AM Gabor Kaszab <gaborkas...@apache.org> > wrote: > > > > Hi Iceberg Community, > > > > It's been a while since we started this discussion. I'd like to revive > the conversion for two reasons: > > 1) I think I'll have some capacity starting from early next year to take > care of the C++ Puffin stuff we already talked about above, and also from > the Impala community we could add some additional auxiliary functionality > for the V3 positional deletes later on. > > 2) I learned that a part of the community is interested in having a C++ > implementation of the Iceberg lib in general for their C++ engine. cc @Gang > Wu > > > > There seemed to be general support from the community to start up such a > sub-project, so I'm reaching out now to ask for some guidance so that we > can get going. @Fokko Driesprong You have much experience in this area, do > you think you have the bandwidth to handhold us with the process and the > steps? Meanwhile, I'll take a look at the Python repo to get a feel for > myself. > > > > Regards, > > Gabor > > > > On Wed, Sep 11, 2024 at 5:32 PM Anton Okolnychyi <aokolnyc...@gmail.com> > wrote: > >> > >> If C++ engines prefer not to depend on Iceberg Rust, I actually don't > see a problem with having a separate C++ project even if it only contains > Puffin readers/writers. The important part is to avoid multiple C++ > writer/reader implementations in different engines. > >> > >> There were concerns with having one repo for multiple languages when we > discussed how to maintain the main project. We should re-evaluate if they > still apply here. > >> > >> - Anton > >> > >> вт, 10 вер. 2024 р. о 06:08 Gabor Kaszab <gaborkas...@apache.org> пише: > >>> > >>> Hey All, > >>> > >>> Thanks for all the answers! > >>> > >>> My initial thought here was to have an iceberg-cpp repo for the Puffin > reader/writer implementation in C++. But then I thought that there might be > only one thing within this cpp repo, that is the Puffin stuff, and once > implemented the repo won't have any more PRs coming in so it might be an > overkill to introduce a separate repo for this purpose. Unless there is a > need from other stakeholders to have a C++ implementation of the Iceberg > lib (similarly to Python and Rust) but this would lead to a different > conversation. > >>> > >>> Pulling in the Rust implementation into the C++ part of Impala: I > don't know, I have to do some research on this. For performance reasons I > know Impala prefers to have its own implementation of things (like Parquet > reader/writer), so I have to double-check if it's acceptable > performance-wise to pull in a Rust implementation of anything. I don't have > any experience of doing so, hence the hesitation. > >>> > >>> JB, you mean that we can create a sub-repo like iceberg-puffin, that > would hold separate implementations for different languages? For me this > could also work. Wondering what others think. > >>> > >>> Regards, > >>> Gabor > >>> > >>> > >>> On Fri, Aug 30, 2024 at 2:08 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >>>> > >>>> Hi Gabor > >>>> > >>>> I like the idea, it sounds good to me. > >>>> > >>>> Imho, for "clarity", the best option would be to have a dedicated > >>>> puffin repo with different language binding (at bit like it's done in > >>>> Arrow). > >>>> I think that adding to iceberg-rust could be an option, but not sure > >>>> it would be obvious for developers. > >>>> > >>>> Regards > >>>> JB > >>>> > >>>> On Thu, Aug 29, 2024 at 2:46 PM Gabor Kaszab <gaborkas...@apache.org> > wrote: > >>>> > > >>>> > Hi Iceberg Community, > >>>> > > >>>> > With the V3 position delete proposal it came up that non-Java > engines might have to implement a Puffin reader and writer themselves so > that they can support the newly proposed position deletes. Impala would > most probably require a C++ implementation and first of all I'm wondering > whether there are other engines that require additional language > implementation. > >>>> > > >>>> > My additional thought is that once we have an implementation in > another language where should that live? I can do the C++ and that could > live within Impala, but I think it could be useful for other engines too > but I have no idea where such an implementation could live TBH. > >>>> > > >>>> > Would be nice to hear opinions on this! > >>>> > Thanks, > >>>> > Gabor >
