Hi all, Based on the last community sync discussion, there were a couple of action items that we wanted to get clarification on for integration of Apache Polaris with Apache XTable to support table formats such as Delta and Apache Hudi.
1. The first item was defining a *Table Converter* Interface, which will have different implementations, such as using XTable or another system for conversion. I have created a draft idea for how this interface can look like here: https://docs.google.com/document/d/15we7EZY9c5mP62WBOIIfaVOG-s4vlPGx1T_SsQeQ150/edit?tab=t.0 2. The second item was around the actual *End to End Flows* for how Polaris communicates with an external XTable REST service. I have created a doc which writes out the conversion flows here: https://docs.google.com/document/d/1SfQXPOc0EtgESEBQPtgn2nTr1lyscxg4t-HskjUSPgI/edit?tab=t.0 <https://docs.google.com/document/d/1SfQXPOc0EtgESEBQPtgn2nTr1lyscxg4t-HskjUSPgI/edit?tab=t.0> Would appreciate the feedback from the community, and hope we can have some time to discuss this in tomorrow's meeting. If not, I'm wondering if we can schedule a dedicated time outside community sync to discuss this proposal more in depth so we can move towards some implementation in the coming weeks. Regards, Rahil Chertara On Thu, Mar 20, 2025 at 9:30 AM Rahil C <rchert...@gmail.com> wrote: > Hi all, > > Just wanted to say thanks to the community for reviewing the > Polaris-XTable proposal and providing helpful thoughts and feedback! Thanks > Eric also for sharing your doc, appreciate that others in the community > were exploring this idea. > > Thanks Yun for bringing up these questions, will try to see if I can put > comments in the xtable proposal doc to share what I am thinking in terms of > the end-to end flow. In terms of the generalized approach I saw that Eric's > doc was defining some interface which would allow for different table > conversion implementations(will leave my thoughts there for that piece > there). > > Happy to discuss in one of the upcoming community syncs, or we can > continue the discussion on the google docs and dev list. > > Regards, > Rahil Chertara > > On Wed, Mar 19, 2025 at 12:17 PM yun zou <yunzou.colost...@gmail.com> > wrote: > >> Hi Rahil, >> >> Thanks a lot for the proposal! It is interesting. As Eric mentioned, we >> were also looking into this before, but at that time there wasn't Generic >> Table support yet. >> >> Now with the coming support for Generic Tables, other than the conversion >> performance, we might also want to look at more into >> 1) How the end to end workflow will look like for users across different >> engines to create tables in one format and load a table in another format >> (given that we have both Iceberg and Generic Table support) ? >> 2) Can the approach be generalized beyond just XTable? XTable is good, but >> users/engines may want to hook up with other available conversions. >> >> Really thank you for bringing this up! I think we should definitely find >> some time to talk about this in more detail. >> >> Best Regards, >> Yun >> >> On Wed, Mar 19, 2025 at 10:23 AM Eric Maynard <eric.w.mayn...@gmail.com> >> wrote: >> >> > Hey Rahil! Thanks for bringing this up. >> > >> > My understanding is that we plan to do this through generic tables. I >> have >> > this old design doc, but I haven't done anything with it yet because >> we're >> > working our way through the initial generic tables implementation: >> > >> > >> https://docs.google.com/document/d/1eZQbwgAx1wzjIYtLIdGg8IKQyw7uR-zN50efJ-Hj5XE/edit?usp=sharing >> > >> > Once we have a better understanding of what generic tables will look >> like, >> > I think we should plan to meet and discuss conversion and the role >> XTable >> > can play in it. >> > >> > --EM >> > >> > On Wed, Mar 19, 2025 at 8:51 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> > wrote: >> > >> > > Hi Rahil >> > > >> > > Welcome ! >> > > >> > > And thanks a lot for your proposal ! It looks very interesting (ok, >> > > I'm a bit biased as Apache XTable mentor :)). >> > > >> > > Let me take a look on the document (and comment directly there). >> > > >> > > Thanks again! >> > > >> > > Regards >> > > JB >> > > >> > > On Wed, Mar 19, 2025 at 4:00 PM Rahil C <rchert...@gmail.com> wrote: >> > > > >> > > > Hi all, >> > > > >> > > > My name is Rahil Chertara, and I’m a part of the Data Infra team at >> > > > Onehouse. >> > > > >> > > > Recently I saw the Roadmap >> > > > <https://github.com/apache/polaris/discussions/1028> for Apache >> > Polaris, >> > > > and became interested in the proposal of Generic Tables and Delta >> > > Support. >> > > > I am interested in understanding the Polaris community's vision for >> > > > supporting open table formats like Delta and Hudi, and was >> wondering if >> > > > Polaris will be handling metadata translation/conversion between >> table >> > > > formats, as mentioned by this overview doc >> > > > < >> > > >> > >> https://docs.google.com/document/d/1H2StuZ26LroibuQni3IJlErlKgrV9fEvYLHHqN7HWfE/edit?tab=t.0#heading=h.b956txtpu769 >> > > > >> > > > ? >> > > > >> > > > In order to assist in this conversation, I wanted to share a >> proposal >> > > with >> > > > the Polaris community on leveraging Apache XTable for handling this >> > > > metadata conversion piece. >> > > > >> > > > Link to my proposal: >> > > > >> > > >> > >> https://docs.google.com/document/d/1gHM9Qco83EFTTAfByyN3hYLLCghuackL5ogC2T1M074/edit?usp=sharing >> > > > >> > > > Appreciate any thoughts and feedback from the community and am also >> > open >> > > to >> > > > discuss this in one of the upcoming community syncs. >> > > > >> > > > Regards, >> > > > Rahil Chertara >> > > >> > >> >
