Re: [DISCUSS] Partial Metadata Loading

Hi Szehon I agree with you there. I think it's better to move forward step by step, so Eduard's proposal is a good idea. However, I think it's worth to keep the discussion going, at least to shape a good proposal. Regards JB On Wed, Nov 6, 2024 at 3:23 AM Szehon Ho wrote: > > There seems to b

Re: [DISCUSS] Partial Metadata Loading

There seems to be many opinions here, but one of the main objections seems to be the complexity added to REST spec impeding newer catalogs. Looking through the actual REST API change proposal

Re: [DISCUSS] Partial Metadata Loading

Hello All, This is an interesting discussion and I'd like to offer my perspective. When a REST Catalog is involved, the metadata is loaded and modified via the catalog API. So control over the metadata is delegated to the catalog. I'd argue that in this situation, catalogs should have the flexib

Re: [DISCUSS] Partial Metadata Loading

Eric, With respect to the credential endpoint, I believe there is important context missing that probably should have been captured in the doc. The credential endpoint is unlike other use cases because the fundamental issue is that refresh is an operation that happens across distributed workers.

Re: [DISCUSS] Partial Metadata Loading

Thanks for this breakdown Dan. I share your concerns about the complexity this might impose on the client. On some of your other notes, I have some thoughts below: Several Apache Polaris (Incubating) committers were in the recent sync on this proposal, so I want to share one perspective related

Re: [DISCUSS] Partial Metadata Loading

I'd like to clarify my concerns here because I think there are more aspects to this than we've captured. *Partial metadata loads adds significant complexity to the protocol* Iceberg metadata is a complicated structure and finding a way to represent how and what we want to piece apart is non-trivia

Re: [DISCUSS] Partial Metadata Loading

Hi Iceberg Community, I just wanted to mention that I was also going to start a discussion about getting partial information from LoadTableResponse through the REST API. My motivation is a bit different here, though: Impala currently has strong integration with HMS and in turn with the HiveCatalog

[DISCUSS] Partial Metadata Loading

Hello Dev list, I want to update the community on the current thread for the proposal "Partially Loading Metadata - LoadTable V2" after hearing more perspectives from the community. In general, there are still some distance to go for a general consensus which I hope to foster more conversations an