Hi all, Just wanted to say thanks to the community for reviewing the Polaris-XTable proposal and providing helpful thoughts and feedback! Thanks Eric also for sharing your doc, appreciate that others in the community were exploring this idea.
Thanks Yun for bringing up these questions, will try to see if I can put comments in the xtable proposal doc to share what I am thinking in terms of the end-to end flow. In terms of the generalized approach I saw that Eric's doc was defining some interface which would allow for different table conversion implementations(will leave my thoughts there for that piece there). Happy to discuss in one of the upcoming community syncs, or we can continue the discussion on the google docs and dev list. Regards, Rahil Chertara On Wed, Mar 19, 2025 at 12:17 PM yun zou <yunzou.colost...@gmail.com> wrote: > Hi Rahil, > > Thanks a lot for the proposal! It is interesting. As Eric mentioned, we > were also looking into this before, but at that time there wasn't Generic > Table support yet. > > Now with the coming support for Generic Tables, other than the conversion > performance, we might also want to look at more into > 1) How the end to end workflow will look like for users across different > engines to create tables in one format and load a table in another format > (given that we have both Iceberg and Generic Table support) ? > 2) Can the approach be generalized beyond just XTable? XTable is good, but > users/engines may want to hook up with other available conversions. > > Really thank you for bringing this up! I think we should definitely find > some time to talk about this in more detail. > > Best Regards, > Yun > > On Wed, Mar 19, 2025 at 10:23 AM Eric Maynard <eric.w.mayn...@gmail.com> > wrote: > > > Hey Rahil! Thanks for bringing this up. > > > > My understanding is that we plan to do this through generic tables. I > have > > this old design doc, but I haven't done anything with it yet because > we're > > working our way through the initial generic tables implementation: > > > > > https://docs.google.com/document/d/1eZQbwgAx1wzjIYtLIdGg8IKQyw7uR-zN50efJ-Hj5XE/edit?usp=sharing > > > > Once we have a better understanding of what generic tables will look > like, > > I think we should plan to meet and discuss conversion and the role XTable > > can play in it. > > > > --EM > > > > On Wed, Mar 19, 2025 at 8:51 AM Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > Hi Rahil > > > > > > Welcome ! > > > > > > And thanks a lot for your proposal ! It looks very interesting (ok, > > > I'm a bit biased as Apache XTable mentor :)). > > > > > > Let me take a look on the document (and comment directly there). > > > > > > Thanks again! > > > > > > Regards > > > JB > > > > > > On Wed, Mar 19, 2025 at 4:00 PM Rahil C <rchert...@gmail.com> wrote: > > > > > > > > Hi all, > > > > > > > > My name is Rahil Chertara, and I’m a part of the Data Infra team at > > > > Onehouse. > > > > > > > > Recently I saw the Roadmap > > > > <https://github.com/apache/polaris/discussions/1028> for Apache > > Polaris, > > > > and became interested in the proposal of Generic Tables and Delta > > > Support. > > > > I am interested in understanding the Polaris community's vision for > > > > supporting open table formats like Delta and Hudi, and was wondering > if > > > > Polaris will be handling metadata translation/conversion between > table > > > > formats, as mentioned by this overview doc > > > > < > > > > > > https://docs.google.com/document/d/1H2StuZ26LroibuQni3IJlErlKgrV9fEvYLHHqN7HWfE/edit?tab=t.0#heading=h.b956txtpu769 > > > > > > > > ? > > > > > > > > In order to assist in this conversation, I wanted to share a proposal > > > with > > > > the Polaris community on leveraging Apache XTable for handling this > > > > metadata conversion piece. > > > > > > > > Link to my proposal: > > > > > > > > > > https://docs.google.com/document/d/1gHM9Qco83EFTTAfByyN3hYLLCghuackL5ogC2T1M074/edit?usp=sharing > > > > > > > > Appreciate any thoughts and feedback from the community and am also > > open > > > to > > > > discuss this in one of the upcoming community syncs. > > > > > > > > Regards, > > > > Rahil Chertara > > > > > >
