Hi Tommy, hi all, Your assistance in testing and evaluating this feature is very welcome!
Here are a few updates on my work: Since the old design doc [1] is mostly out of date, I'm currently exploring two design choices for authorizing sign requests that we briefly discussed, but never settled on: 1. A signing token, issued when the table is loaded and validated by the sign endpoint; this only works with Iceberg 1.12+. This is similar to how Project Nessie does remote signing. In this mode, the authorization decision is expected to be fast, because it is pre-computed in the token. 2. Per-request authorization, with or without a short-lived decision cache; this works for any Iceberg client. In this mode, authorization is done the normal way, which may incur in calls to the database and/or to the external authorizer, if any. The decision cache is a small enhancement where an authorization decision is cached for 30 seconds by default. To compare them, I created a new Gatling benchmark for remote signing [2], and I'm collecting preliminary results by running it locally against my working branch. A few early numbers (single machine, Polaris capped at 4 CPUs, 1000 sign requests per table load, so take them as indicative only), in max sign requests/s per node: - JDBC: token 12.7k, per-request without cache 10.3k (-19%), per-request with cache 12.4k - NoSQL: token 11.8k, per-request without cache 4.8k (-60%), per-request with cache 12.0k Using per-request authz without a decision cache is clearly the worst scenario, as expected: the gap is mostly due to entity resolution costs, and also cache deserialization for the NoSQL backend. I also tested external principals (credential-mode=external) and OPA: the numbers are similar or slightly worse due to repeated calls to OPA. However, per-request authz with a decision cache is roughly on par with signing tokens. That's an interesting observation and begs the question: which method should Polaris implement? Both? Each approach has pros and cons: - Signing tokens: - Pros: works out of the box with staged table creations: we capture the authz decision at load time - Cons: tokens are valid for X hours, and cannot be revoked or refreshed (at least not for now - maybe later) - Per-request authz with decision cache: - Pros: cache is short-lived (a few seconds) compared to token lifetime - Cons: does not work well with staged table creations (the sign request arrives before the table commit) I am inclined to use a mix of both: per-request authz by default, except for staged table creations where we'd use signing tokens. What are your thoughts? Thanks, Alex [1]: https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing [2]: https://github.com/apache/polaris-tools/pull/304 On Mon, Nov 2, 2026 at 9:27 PM Tommy Bollhofer <[email protected]> wrote: > > Hi Alex, > > Thank you for the update, and for linking the original thread. > > Great to hear that you're planning to revisit the previous PR. We'd be happy > to test an experimental build against our on-premises FlashBlade > S3-compatible storage and share our findings, including compatibility across > the various engines and versions we're currently using. > > Our client stack includes Trino with Iceberg 1.12, Spark with the Iceberg > 1.10 runtime, PyIceberg, and DuckDB. Given the relatively short duration of > our workloads, we should be able to begin testing without requiring > credential refresh support. > > Please let us know when there's a build available to evaluate. We'd be glad > to help validate the implementation and provide feedback. > > Thanks again for your work on this! > > Best, > Tommy > > > ________________________________________ > From: Alex Dutra <[email protected]> > Sent: Tuesday, October 27, 2026 13:19 > To: [email protected] <[email protected]> > Cc: Tommy Bollhofer <[email protected]> > Subject: Re: [DISCUSS] Resume remote signing work > > > Hi Tommy, > > Thanks for chiming in! Your reply got disconnected from the thread though, so > here is the link to my first message just in case: > > https://lists.apache.org/thread.html/vk5zj7qtrztl36c9c1qt6d09lryjltdk > > The details you provide are very valuable: since FlashBlade doesn't support > AssumeRole, remote signing becomes indeed a good alternative. And since you > can live for now without auth-credentials refresh, it seems to me we have > enough evidence that we should move forward and provide a first, experimental > implementation of remote signing. > > I will start preparing a revisited PR, built on top of the old one. > > Thanks, > Alex > > > > > On Thu, Oct 22, 2026 at 7:06 PM Tommy Bollhofer via dev > <[email protected]> wrote: > Hi all, > > +1 to resuming the remote signing work now, rather than waiting for the > spec on refreshing auth data. Alex asked on the issue (#32) for our use > case, so here it is. > > We run Polaris against an on-prem S3-compatible store (a FlashBlade). > Its STS support is limited to AssumeRoleWithWebIdentity, so Polaris > cannot vend credentials through its current AssumeRole flow. As a > result we turn vending off and distribute static storage keys to every > service in every environment: Polaris, Trino, Spark (table maintenance), > and PyIceberg and DuckDB clients. That is a rotation and blast-radius > burden that grows with each environment we add. Remote signing would > remove the static keys from the clients, which is why we would like to > see it move. > > On the auth-refresh gap: our workloads are mostly short queries and > pipeline runs, and our longest jobs are Spark maintenance operations > that take minutes, so as far as we can tell we could start without > refresh support and pick it up incrementally, as Alex suggested. > > Client versions are mixed on our side (Trino on Iceberg 1.12, Spark on > the Iceberg 1.10 runtime, PyIceberg and DuckDB with their own Iceberg > implementations), so we expect client-side support to vary. We would be > happy to test early builds against FlashBlade and report which engines > and versions work. > > (On #32 we also described a smaller option, AssumeRoleWithWebIdentity > as an alternative STS flow. Remote signing is the fuller fix and the > one we would rather see.) > > Best, > Tommy > > On 2026-10-02 15:00 UTC Alex Dutra wrote: > > Hi all, > > > > Polaris will soon be updated to Iceberg 1.12. > > > > This release includes the new remote-signing-config field within > > LoadTableResult, which addresses nearly all remaining requirements for > > Polaris to complete its remote signing implementation, abandoned a > > while ago due to impending spec changes. > > > > However, one detail remains: when we added the remote-signing-config > > structure, refreshing the contained authentication or authorization > > data was intentionally left out of scope. I plan to start a separate > > discussion and spec change proposal with the Iceberg community > > regarding this shortly. > > > > Despite this, the core remote signing functionality is again viable in > > Polaris today, with Iceberg 1.12. The only missing piece being that > > auth data cannot be refreshed yet, though this capability could be > > added incrementally in a future update. > > > > With that in mind, I would like to ask the Polaris community: > > > > Should we resume the remote signing work where we left [1]? Or would > > it be preferable to wait until the specification for refreshing auth > > data is finalized? > > > > For additional background, you can refer to the original DISCUSS > > thread [2], but keep in mind that much of it is now outdated. > > > > Thanks, > > Alex > > > > [1]: https://github.com/apache/polaris/pull/3995 > > [2]: https://lists.apache.org/thread.html/xpf60ko823c7mlg5bhnddq3btvdsq5s1 > > > > Tommy Bollhofer > Senior Data Engineer > Voloridge.com > [email protected] > 561.231.5770 > > This email and any attachments are intended only for the intended > recipient(s) and may contain information that is privileged and/or > confidential. If you are not an intended recipient, or have received this > email in error, please notify the sender immediately and permanently delete > this email, its attachments, and any copies. Any disclosure, copying, > distribution, or reliance on the contents by anyone other than an intended > recipient is unauthorized. This email message (including any attachments) is > not to be construed as investment advice or the solicitation of an offer to > buy or sell interests in any securities. Any information provided in this > message must be considered in conjunction with the appropriate disclosure > documents. PAST PERFORMANCE IS NOT INDICATIVE OF FUTURE RESULTS.
