Hi All, Following up on the ongoing discussion thread regarding the file-level access delegation modes in the Iceberg REST spec, I wanted to share some findings from a cross-cloud perspective of how the major Cloud Service Providers handle these protocols.
When evaluating remote signing as an access mode, two major operational hurdles emerge: strict 15-minute Time-To-Live constraints and a lack of native support across all CSPs, particularly Azure. 1. Non-Configurable TTLs AWS: In the AWS SigV4 protocol, the calculated signature with a timestamp inherently expires 15 minutes after. [1] GCS: GCS enforces an identical constraint for remote header signing. A signature passed via request headers is valid only from 15 minutes prior to the active datetime until exactly 15 minutes after. [2] 2. Lack of Native Remote Signing Support in Azure While AWS (SigV4) and GCS (V4) natively accept short-lived, granular signatures via the Authorization header, Microsoft Azure Blob Storage and ADLS Gen2 fundamentally reject this paradigm. In Azure Storage, a Shared Access Signature (SAS) is strictly a token designed to be appended to the resource URI as query parameters. [3] To execute true remote header signing for Azure Storage, a catalog would be forced to use the legacy SharedKey authorization scheme, formatting the header as Authorization: SharedKey <AccountName>:<Signature>. Constructing this signature mathematically requires the catalog to hold the storage account's master access key. [4] [5] Because of this, Microsoft explicitly recommends disallowing SharedKey authorization entirely, advising modern architectures to rely instead on User Delegation SAS tokens (via the URI) or Microsoft Entra ID. [5] On the other hand, pre-signed URLs are natively and uniformly supported across all three major CSPs. While the underlying cryptographic signatures and generation methods differ (AWS SigV4, GCS V4, Azure SAS), the resulting access artifact is universally a standard, fetchable HTTPS URL. Furthermore, pre-signed URLs allow for highly customizable expiration times. The maximum TTL is upper bound by the validity of the signing credential itself (e.g. temporary STS credentials which are default 1-hour sessions but can be configured up to 12 hours). This provides significantly more operational flexibility than the rigid 15-minute window enforced on header-based signatures. Given the findings above, I would still like to discuss the possibility of pre-signed URLs. During our last sync, a valid concern was raised: our current SDKs and FileIO implementations don't support processing raw, pre-signed HTTPS URLs. Thus requiring a new, separate read path to be built. To help us evaluate exactly what this effort entails and to discuss the trade-offs, I’ve put together a POC for what this pre-signed URL read path might look like: https://github.com/williamhyun/iceberg/pull/206 Looking forward to hearing your thoughts! Best, William [1] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-request-authentication.html#sqs-api-basic-authentication-process [2] https://docs.cloud.google.com/storage/docs/authentication/signatures#active_datetime [3] https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview#how-a-shared-access-signature-works [4] https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#specifying-the-authorization-header [5] https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access?tabs=blobs#authorization-for-data-operations On 2026/07/08 01:16:35 William Hyun wrote: > Hi all, > > Following up on the earlier discussion around file-level access delegation > in the REST Catalog spec, I put together a small POC PR against my personal > Iceberg fork to explore the remote signing + caching direction we discussed: > > https://github.com/williamhyun/iceberg/pull/198 > > The goal is to make the tradeoffs more concrete and evaluate whether this > could be a practical approach for finer-grained delegated access. > Feedback is very welcome. > > Best, > William > > > > On Thu, Jul 2, 2026 at 5:29 PM William Hyun <[email protected]> wrote: > > > Hi all, > > > > Thank you again for the discussion during the 7/1 catalog community sync. > > I wanted to summarize two alternative directions that came up to > > pre-signed URLs: > > 1. scoped credential vending > > 2. remote signing with caching and bulk-signing > > > > I think scoped credential vending is worth considering in cases where the > > authorization boundary maps cleanly to a small number of stable storage > > prefixes. > > Scoped credential vending seems less practical when access needs to be > > granted for many specific files rather than a small number of prefixes. > > > > In AWS, session policies are passed as parameters during AssumeRole-style > > issuance when the temporary session is created. [1] > > This means that scoping down an already-issued credential cannot be done > > locally by simply altering the existing token. > > Instead, it requires minting another STS session whose policy encodes the > > new scope. > > This turns authorization into an STS issuance problem with limitations on > > STS request frequency [2] and policy size. [3] > > > > By contrast, pre-signed URLs and remote signing can authorize exact object > > requests using already-held credentials, without creating a new STS session > > for each file set. > > Based on that discussion, I am exploring a POC around bulk remote > > signing + caching to evaluate whether it could be a practical alternative > > to pre-signed URLs for these finer-grained access patterns. > > > > Best, > > William > > > > [1] https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html > > [2] > > https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html#reference_iam-quotas-sts-requests > > [3] > > https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html > > > > On Tue, Jun 30, 2026 at 7:43 PM William Hyun <[email protected]> wrote: > > > >> Hi everyone, > >> > >> I'd like to open a discussion on an extension to the REST Catalog spec to > >> enable file-level access delegation during scan planning. > >> Today, delegated access is table-scoped, which can force > >> over-provisioning or table fragmentation when consumers should only see a > >> subset of partitions or records. > >> This proposal adds a path using pre-signed URLs to make partition-scoped > >> sharing practical in the REST Catalog model. > >> > >> Here is my proposal: https://s.apache.org/n16st > >> > >> I look forward to hearing your thoughts. > >> > >> Relevant discussion: > >> - https://lists.apache.org/thread/ko9kp0gvzhx85n7cvoxqnpw4vwnhmdg6 > >> > >> Best, > >> William > >> > >> >
