Hey Yufei, for 1) a client would choose the longest matching prefix. In terms of failure I guess it really depends on what kind of credentials the server sent to the client. If the server sent multiple credentials for the same table (one generic (*prefix=s3*) and one narrowed-down one ( *prefix=s3://my-custom-bucket/...*)) and the client accidentally chose the generic one, then it really depends on the underlying credential whether the client fails or not.
For 2) I don't see any good reason to introduce any kind of versioning for this. It's worth mentioning that we're planning to document storage-specific configuration with #10576 <https://github.com/apache/iceberg/pull/10576>, so that should generally solve this issue and servers/clients should know what to send/use. Eduard On Mon, Oct 14, 2024 at 7:34 PM Yufei Gu <flyrain...@gmail.com> wrote: > Hi Eduard, > > Thanks for the proposal. I'm excited about the new spec. I have two > questions: > > 1. This is probably a dumb question due to the lack of context, but I'm a > bit confused about how clients should select a prefix to use. In scenarios > where multiple prefixes exist, which one should the clients choose? If a > client selects the wrong prefix, what would be the implications? > 2. The proposal suggests a completely schema-less approach for handling > credentials using key-value pairs. How do we manage potential > incompatibilities when the credential schema changes, such as adding a new > key or renaming an existing one? The current design relies entirely on > synchronization between the server and client implementations. While > credential properties tend to be stable, changes can still occur. Ignoring > this risk could result in client failures when server-side changes are > made, especially since these credentials aren't versioned like API > endpoints (e.g., v1, v2). Should we consider introducing versioning to > mitigate this risk? > > > Yufei > > > On Thu, Oct 10, 2024 at 2:02 PM Jack Ye <yezhao...@gmail.com> wrote: > >> +1 for adding this in the REST spec. >> >> Glue has a similar API GetTemporaryGlueTableCredentials [1], which was >> introduced because of performance and also security reasons. For example, >> we don't want to propagate credentials across the compute nodes in the >> cluster, and each compute node needs to fetch only the credentials >> independently. Such an API becomes handy to do improvements like caching. >> >> Best, >> Jack Ye >> >> [1] >> https://docs.aws.amazon.com/cli/latest/reference/lakeformation/get-temporary-glue-table-credentials.html >> >> >> On Thu, Oct 10, 2024 at 3:47 AM Eduard Tudenhöfner < >> etudenhoef...@apache.org> wrote: >> >>> Hey everyone, >>> >>> I'd like to propose a mechanism and changes in order to be able to >>> refresh vended credentials for tables. >>> >>> Please find the proposal doc here >>> <https://docs.google.com/document/d/1acCkaPCO7WsLtvYugrayurbef4zCnD2rb3ZPBKeaYoo/edit?usp=sharing> >>> . >>> The proposal requires a spec change, which can be seen in #11281 >>> <https://github.com/apache/iceberg/pull/11281>. >>> >>> As discussed in the last sync, this should hopefully help in better >>> understanding the proposal around standardizing credentials in the OpenAPI >>> spec, which is being discussed in >>> https://lists.apache.org/thread/jmklpnywnghg7qwmwr14zj2k6tnxmdo4. >>> >>> Thanks, >>> Eduard >>> >>
