Hi Rahil, Thanks a lot for the proposal! It is interesting. As Eric mentioned, we were also looking into this before, but at that time there wasn't Generic Table support yet.
Now with the coming support for Generic Tables, other than the conversion performance, we might also want to look at more into 1) How the end to end workflow will look like for users across different engines to create tables in one format and load a table in another format (given that we have both Iceberg and Generic Table support) ? 2) Can the approach be generalized beyond just XTable? XTable is good, but users/engines may want to hook up with other available conversions. Really thank you for bringing this up! I think we should definitely find some time to talk about this in more detail. Best Regards, Yun On Wed, Mar 19, 2025 at 10:23 AM Eric Maynard <eric.w.mayn...@gmail.com> wrote: > Hey Rahil! Thanks for bringing this up. > > My understanding is that we plan to do this through generic tables. I have > this old design doc, but I haven't done anything with it yet because we're > working our way through the initial generic tables implementation: > > https://docs.google.com/document/d/1eZQbwgAx1wzjIYtLIdGg8IKQyw7uR-zN50efJ-Hj5XE/edit?usp=sharing > > Once we have a better understanding of what generic tables will look like, > I think we should plan to meet and discuss conversion and the role XTable > can play in it. > > --EM > > On Wed, Mar 19, 2025 at 8:51 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Hi Rahil > > > > Welcome ! > > > > And thanks a lot for your proposal ! It looks very interesting (ok, > > I'm a bit biased as Apache XTable mentor :)). > > > > Let me take a look on the document (and comment directly there). > > > > Thanks again! > > > > Regards > > JB > > > > On Wed, Mar 19, 2025 at 4:00 PM Rahil C <rchert...@gmail.com> wrote: > > > > > > Hi all, > > > > > > My name is Rahil Chertara, and I’m a part of the Data Infra team at > > > Onehouse. > > > > > > Recently I saw the Roadmap > > > <https://github.com/apache/polaris/discussions/1028> for Apache > Polaris, > > > and became interested in the proposal of Generic Tables and Delta > > Support. > > > I am interested in understanding the Polaris community's vision for > > > supporting open table formats like Delta and Hudi, and was wondering if > > > Polaris will be handling metadata translation/conversion between table > > > formats, as mentioned by this overview doc > > > < > > > https://docs.google.com/document/d/1H2StuZ26LroibuQni3IJlErlKgrV9fEvYLHHqN7HWfE/edit?tab=t.0#heading=h.b956txtpu769 > > > > > > ? > > > > > > In order to assist in this conversation, I wanted to share a proposal > > with > > > the Polaris community on leveraging Apache XTable for handling this > > > metadata conversion piece. > > > > > > Link to my proposal: > > > > > > https://docs.google.com/document/d/1gHM9Qco83EFTTAfByyN3hYLLCghuackL5ogC2T1M074/edit?usp=sharing > > > > > > Appreciate any thoughts and feedback from the community and am also > open > > to > > > discuss this in one of the upcoming community syncs. > > > > > > Regards, > > > Rahil Chertara > > >
