Hey Dan, Thanks for taking a look at this!
Regarding the server returning signed urls being slightly different from supporting pre-signed being slightly different problems, I have a slightly different POV here (I know we discussed a bit offline as well just wanted to share our thoughts here as well, as how we have approached this problem so far) . Because how else are we planning to refresh these vended urls if that expires before consumption ? Scan planning and execution i.e reading these files happen in two different phases of these engines. Typically a spark driver would plan it and executor would consume it, it may happen that the url expires before it is consumed by the executor. In this case we need to have a refresh mechanism like we do for the vended creds like /credentials endpoint [1], there is no requirement to call it only when your creds expire to refresh it in the protocol, a client can directly call it if it wants creds. Hence the design had both aspects to make it a complete solution [2] otherwise this is not usable for practically long running queries please check this out check practical example here [3]. And we need to keep that simple otherwise we would need very heavy machinery to keep them in sync and ask the server to re-plan. Orthogonally I would additionally, also recommend passing the planId to the pre-sign call for auditability POV like i mentioned in the spec PR [4] and also advocate for the need for batch sign at-least which could be leveraged at an executor level need not be at a task level [5], like we do we do for vended creds. Regarding repurposing the sign vs creating the new endpoint I think it's fine if we want to re-purpose the sign endpoint, the proposal had both [6] and we initially had approach 1 as the presign dedicate endpoint because existing request / response model is already called remote signing which is already well defined term in industry, also it helps in discoverability pov as from /config we can return this feature is supported or not, i understand we want to say remote sign in the context the server is doing is doing signing part (which can be further remote-sign / pre-sign)... I was trying to narrow down such requirements before making the changes. I agree we should align on the implementation here and avoid any duplicate effort especially when there is active work / discussion going on, Let's work together to get this in, I will check the PRs to find a path forward ! Now talking about FILE_TYE datatype I am unsure how we are mixing FILE level access with the FILE_TYPE support which is still in talks, sure we can repurpose this machinery but they are not the same ? I have tons of questions on this front specially with FILE_TYPE uri ownership from table pov, I would suggest concluding on them before making concrete assumptions, I agree we should leverage this thread / community sync to discuss this more. Thanks Sung for starting this, a couple of questions I would like to contribute to this thread that comes on top of my mind, happy to discuss this in sync as well: 1. Do we always want to give FILE_TYPE pointing uri access only via pre-sign ? 2. If the file.uri doesn't belong to the table prefix ... how would the server know this file.uri belongs to the table (as this is embedded deep within a row of table) to authorize the access ? does having this requirement Is forcing the server to keep a very expensive trie index in its working state ? or are we intentionally trying to be less restrictive from spec pov 3. How will the server orchestrate the way of reading data files via vended creds and FILE_TYPE via pre-signed urls if we go with #1, some sign config or table level metadata ? presently it's either do vended creds only or remote signing only. 3. Can FILE_TYPE access also use vended creds if yes how does the server orchestrate that ? Looking forward to the discussion ahead / FILE_TYPE syncs. Best, Prashant Singh [1] https://github.com/apache/iceberg/commit/35a02d035e40344523fdb7a0933e07a8433ea763 [2] https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?disco=AAACCbiQWNk [3] https://github.com/delta-io/delta-sharing/issues/941 [4] https://github.com/apache/iceberg/pull/17332#discussion_r3992567118 [5] https://github.com/williamhyun/iceberg/pull/206/changes#diff-7347bb69aaf4647d472cb8d0f3aedfcdff3158a0a9d23aa0e050c5fc180e304fR31 [6] https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?tab=t.ez1lkov8k91#heading=h.vjmmhe69p5j3 <https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?tab=t.ez1lkov8k91#heading=h.vjmmhe69p5j3> On Wed, Sep 16, 2026 at 12:11 AM Daniel Weeks <[email protected]> wrote: > I left comments on both sets of PRs and agree we need to align the > implementations. > > The work for presigned urls via plan/tasks is slightly different from > remote pre-signing. Plan/tasks urls are assumed to already be presigned > when handed back, so the IO implementations only need to detect and execute > the request. Remote pre-signing is a step prior to this where a storage > native path (e.g. s3://...) is routed to the catalog for pre-signing prior > to opening a stream (via the same path as scan/plan). > > I think we should get remote-presigning into a state where it's a viable > alternative to the current remote signing (largely to address Azure's > limitations on remote signing) and this should leverage the same > infrastructure for plan/tasks. (Also, this does not require introducing a > /presign endpoint). > > Once pre-signing is in place, we can leverage it as a foundation for some > of the file based access delegation. I know there is some PoC work in that > area (bulk signing), but I don't agree with the direction that's going and > we need to align on how to introduce bulk operations in a way that works > for File delegation. > > We'll discuss more at the File sync, but much of this is common across all > of these use cases and we need to introduce the implemention in a way that > aligns with all of the various use cases. > > -Dan > > On Wed, Sep 16, 2026 at 3:12 AM Sung Yun <[email protected]> wrote: > >> Thanks Prashant, that makes sense. I’m glad to hear you are carrying >> William’s work forward. >> >> If the plan is still to introduce /presign for the plan API response, go >> ahead; I will subscribe to the PR discussion and fold in what comes out of >> this thread as the use case discussion evolves. I agree that we can pool >> our efforts here. >> >> As noted in the FILE proposal feedback, and in the review comments on >> William's PR where we began discussing FILE governance, I think it will >> help to keep this thread on that use case. What file column usage patterns >> need, and what SDK or spec changes follow from it. >> >> Sung >> >> On 2026/09/15 21:33:52 Prashant Singh wrote: >> > Thanks for starting this thread, Sung! >> > >> > I wanted to share some additional background on the work already >> underway >> > in this area. William and I opened a spec proposal about three months >> ago >> > focused on supporting file-level access through pre-signed URLs [1]. The >> > initial goal was to enable partition-level sharing (with one file >> belonging >> > to a single partition). >> > >> > We went through several POCs [2] and a number of community discussions >> to >> > evaluate whether the existing remote-signing and batch remote-signing >> > approaches would be sufficient. In particular, the Azure POC [3] helped >> > highlight some of the limitations of remote signing across cloud >> providers >> > and provided useful evidence for considering pre-signed URLs as an >> > additional access-delegation mechanism. >> > >> > The work has two main pieces: SDK support for the /plan endpoint to >> return >> > pre-signed URLs [4], and support for obtaining/refreshing pre-signed >> URLs >> > through the signing path [5][6], along with the associated POC [7]. >> > >> > I noticed that you have opened PRs #18080 and #18110 for FILE-type >> access. >> > The foundational work from the existing PRs, including returning >> pre-signed >> > URLs from the plan API and adding client-side support, should be >> directly >> > reusable for the FILE work. Given the amount of discussion and >> > implementation already underway, I'd love to collaborate on building on >> > these existing tracks rather than creating parallel implementations. >> > >> > Happy to work through this together and make sure the existing work can >> > support the FILE use cases you outlined as well. >> > >> > Best, >> > Prashant >> > >> > [1] https://lists.apache.org/thread/odof6m2npvktwd51cz8qnrxjv95ws4wm >> > [2] >> > >> https://github.com/williamhyun/iceberg/pulls?q=is%3Apr+state%3Aopen+author%3Awilliamhyun+sign >> > [3] https://github.com/williamhyun/Azure-SAS-Header-POC >> > [4] https://github.com/apache/iceberg/pull/17457 >> > [5] https://github.com/apache/iceberg/pull/17332 >> > [6] >> > >> https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?tab=t.ez1lkov8k91#heading=h.vjmmhe69p5j3 >> > [7] https://github.com/williamhyun/iceberg/pull/206 >> > >> > On Mon, Sep 14, 2026 at 3:53 PM Sung Yun <[email protected]> wrote: >> > >> > > Hi folks, >> > > >> > > >> > > There is already work underway to return pre-signed URLs from a plan >> API >> > > response, which I believe William and Prashant have scoped to support >> that >> > > user journey [1]. As I was reviewing that work, I began thinking ahead >> > > about delegated access patterns for the ongoing FILE type [2] work, >> and as >> > > Dan mentioned in this comment [3], we have yet to have a dedicated >> > > discussion on access delegation for FILE. So I wanted to start this >> thread >> > > to begin that discussion. >> > > >> > > >> > > Today access is delegated through remote signing and vended >> credentials, >> > > and I believe both can support FILE type access delegation use cases >> as >> > > long as the object is read by a storage client. With multimodal >> access, >> > > the interesting work often happens in other services, for example an >> > > inference service, that need to retrieve the objects to run the >> inference >> > > itself, the result of which is then returned to the calling process >> for >> > > further analytics. While we could require that these functions run in >> the >> > > calling process, or that those services act as storage clients >> themselves, >> > > I think enabling FILE reference URLs to be signed so that they are >> > > optimized for sharing with other services will help Iceberg's FILE >> > > integration plug readily into the existing multimodal ecosystem. >> > > >> > > >> > > While work is still underway on the FILE type discussions to finalize >> the >> > > shape of the Iceberg type, I'd like to start the discussion on what >> access >> > > enhancements would need to be added to the Iceberg REST Catalog spec >> and >> > > its supporting SDKs. I've put together a spec PR based on recent >> catalog >> > > community discussion [4] on pre-signed urls, along with a draft PR >> that >> > > POCs the integration on the client side [5]. >> > > >> > > >> > > I'd love to get the community's thoughts on whether client initiated >> > > pre-signing + sharing URLs is an additional mode we want to introduce >> to >> > > enable FILE access sharing as we look to integrate the Iceberg client >> into >> > > the multimodal ecosystem. >> > > >> > > >> > > I'd also be interested to hear what other consumer patterns people are >> > > running into beyond integrating with an inference service, which is >> the >> > > case I've been designing against. >> > > >> > > >> > > Sung >> > > >> > > >> > > [1] https://github.com/apache/iceberg/pull/17457 >> > > >> > > [2] https://s.apache.org/iceberg-fileref >> > > >> > > [3] >> https://github.com/apache/iceberg/pull/17332#discussion_r3992139348 >> > > >> > > [4] https://github.com/apache/iceberg/pull/18080 >> > > >> > > [5] https://github.com/apache/iceberg/pull/18110 >> > > >> > >> >
