Re: [PROPOSE] Disable credential vending for external catalogs by default

I do like the simplicity. Unfortunately, I don’t think it meets the “secure by default” test. An admin may enable unstructured locations without understanding the credential vending implications. Now, in your scenario, if the admin enabled unstructured table locations and that implicitly turned off

Re: [PROPOSE] Disable credential vending for external catalogs by default

I see. I am imagining a scenario like the following: I'm a Polaris admin who wants to add a new external catalog that points to an existing Glue catalog. Since I know the tables in that catalog occupy all sorts of wacky locations, I specify *ALLOW_UNSTRUCTURED_TABLE_LOCATION* as true when I create

Re: [PROPOSE] Disable credential vending for external catalogs by default

Admins don’t have the knob because they don’t control the structure of the remote catalog. This could be something like Glue or Tabular that generate UUIDs for their table locations. Or it can be entirely user controller where tables are randomly scattered across the bucket. E.g., maybe the user wa

Re: [PROPOSE] Disable credential vending for external catalogs by default

> They do not have such a knob for EXTERNAL catalogs This by itself seems like a problem to me. It seems like we do check

Re: [PROPOSE] Disable credential vending for external catalogs by default

I responded to your comment on the PR already. I think a flag to disable credential vending for INTERNAL catalogs does make sense, but I think it should be a separate flag. Service admins already have the knobs to make an INTERNAL catalog secure by default by enforcing non-overlapping table locati

Re: [PROPOSE] Disable credential vending for external catalogs by default

I commented the same on the PR, but it’s not obvious to me why we would make this exclusive to external catalogs. You can create internal catalogs with overlapping locations too. A single flag to disable credential vending altogether solves this problem across the board and also allows for finer-g

[PROPOSE] Disable credential vending for external catalogs by default

Hey folks I opened a PR at https://github.com/apache/polaris/pull/395 to support disabling credential vending for external catalogs. Currently, some remote catalogs don't control the table location for Iceberg tables, allowing people to create tables in overlapping directories in storage. A perso

Re: Heading to Polaris 0.9 release and beyond

Good point. Imho, we should increment API versions when we introduce "breaking changes". On major versions, I don't see a need to support previous API versions. Users who want to support previous API can always use previous Polaris versions. For us, we can "maintain" two major versions in parallel