Re: [PROPOSAL] Partially Loading Metadata - LoadTable V2

2024-10-10 Thread Haizhou Zhao
Thanks Eduard and Dan, At this stage, my main goal is to check around the community whether this problem is worth solving. If I can get sufficient feedback, or better, even consensus from the community, then that lays down a good foundation to further progress this thread. Implementation details a

Re: [PROPOSAL] Partially Loading Metadata - LoadTable V2

2024-10-10 Thread Daniel Weeks
Hey Haizhou, I think you've done a great job of capturing some of the metadata size related issues in the doc, but I would echo Eduard's comments that we should explore using the existing refs only loading first. This may require adding similar functionality for schemas/logs if we think that is a

Re: [PROPOSAL] Partially Loading Metadata - LoadTable V2

2024-10-10 Thread Eduard Tudenhöfner
Hey Haizhou, thanks for working on that proposal. I think my main concern with the current proposal is that it adds quite a lot of complexity at a bunch of places, since you'd need to partially update *TableMetadata*. Additionally, it requires a new endpoint. An alternative to that would be to do