Re: [PROPOSAL] Partially Loading Metadata - LoadTable V2

2024-10-10 Thread Haizhou Zhao
Thanks Eduard and Dan, At this stage, my main goal is to check around the community whether this problem is worth solving. If I can get sufficient feedback, or better, even consensus from the community, then that lays down a good foundation to further progress this thread. Implementation details a

Re: [PROPOSAL] Partially Loading Metadata - LoadTable V2

2024-10-10 Thread Daniel Weeks
Hey Haizhou, I think you've done a great job of capturing some of the metadata size related issues in the doc, but I would echo Eduard's comments that we should explore using the existing refs only loading first. This may require adding similar functionality for schemas/logs if we think that is a

Re: [PROPOSAL] Partially Loading Metadata - LoadTable V2

2024-10-10 Thread Eduard Tudenhöfner
Hey Haizhou, thanks for working on that proposal. I think my main concern with the current proposal is that it adds quite a lot of complexity at a bunch of places, since you'd need to partially update *TableMetadata*. Additionally, it requires a new endpoint. An alternative to that would be to do

[PROPOSAL] Partially Loading Metadata - LoadTable V2

2024-10-09 Thread Haizhou Zhao
Hello Dev List, I want to bring this proposal to discussion: https://docs.google.com/document/d/1eXnT0ZiFvdm_Zvk6fLGT_UxVWO-HsiqVywqu1Uk8s7E/edit#heading=h.uad1lm906wz4 It proposes a new LoadTable API (branded LoadTableV2 at the moment) on REST spec that allows partially loading table metada