Thanks Prashant for your reply. I don't think the proposal took an orthogonal approach, but the submitted proposal added more than just support for Iceberg expressions, which is something some members have been hesitant about. We may have taken a bit more time compared to what was expected in incorporating the changes but we also wanted to update some of our prototypes and validate the overall approach. We did reply promptly to your email and offered several areas of collaboration around Iceberg expression, and we may not have acknowledged fully the list of changes you have been pushing for, but not acknowledging to do something is not quite the same as acknowledging not to do something.
But to be honest, I don't think this is worth rehashing and what matters is to realign the various people interested in this proposal. My original question still stands: what's the correct process to follow here? should the doc be the source of truth and PR updated based on doc evolution, or is the PR now the source of truth for the proposal? Both places do have comments from different people and have different elements of context/research. Laurent On Mon, Sep 29, 2025 at 2:46 PM Prashant Singh <prashant010...@gmail.com> wrote: > Since I am the author of the PR being referenced. I would like to provide > some context regarding the FGAC proposal. > > An initial high-level FGAC proposal [1] was made to the community. The > discussions led to a suggestion that we move directly to a spec change > proposal—similar to what was done for events—since returning access > decisions had already been discussed in the community extensively in the > past[2]. > > A different FGAC proposal [3] was then presented, but it took an entirely > orthogonal approach (using SQL strings rather than just Iceberg > expressions), even though PMC members had suggested sticking with Iceberg > expressions[4]. > > In community sync discussions [5], there was consensus to move forward > with only Iceberg expressions—highlighting their extensibility, > particularly when SQL UDFs are introduced alongside loadTable returning > read restrictions. > > After about a month without progress, I bumped the email thread to check > in. I also offered help in case the author was busy, since this is a > community-driven effort. The author hasn’t acknowledged the willingness to > change the design based on the community feedback on using Iceberg > expressions and reusing loadTable API. So I created the spec draft PR [6] > to illustrate the approach with broad community consensus and help > facilitate the discussion. That iceberg PR was linked [7] in the same > thread so that both the author and the broader audience were aware. The > draft PR was discussed openly in public forums of community syncs and the > Iceberg repo [8]. > > A month later, the author updated their original proposal [9] to adopt > Iceberg expressions. But it still did not address the one key > feedback—namely, that restrictions should be returned as part of loadTable, > similar to how credentials are handled, rather than introducing a new > endpoint. > > I hope this clarification helps fill in any context that may have been > missed. > > Best, > > Prashant Singh > > > > [1]https://lists.apache.org/thread/nfw1t0glfdfj1hwmzzzzwwyrfnq44yr5 > [2] > https://docs.google.com/document/d/14nmuxxfzQsYo59o0Fbpb-pxOlzS6bVtduL8P8pwKZ6U/edit?tab=t.0#heading=h.cuvuc7yf52ry > [3] > https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.0#heading=h.p7irdnqk4sk6 > [4] https://lists.apache.org/thread/blsvzx3y62kpyhob587tmhmlps6v6jho > [5] https://www.youtube.com/watch?v=RRyohCUDnME > [6]https://github.com/apache/iceberg/pull/13879 > [7] https://lists.apache.org/thread/8t2zh9nchklm4zwjj89vnq9fg9wv45o4 > [8] https://www.youtube.com/watch?v=orAXA5e9pmU > [9] > https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38#heading=h.p7irdnqk4sk6 > > On Mon, Sep 29, 2025 at 9:36 AM Laurent Goujon <laur...@dremio.com.invalid> > wrote: > >> Hi, >> >> I’m trying to clarify what is the process regarding discussing proposals >> in general but also in the context of the Iceberg REST FGAC one. >> >> While there’s engagement on the mailing list and updates to the document >> being made, a parallel conversation is happening on a pull request directly >> (https://github.com/apache/iceberg/pull/13879/files). >> >> I've assumed that proposals were to be discussed first on the mailing >> list using google docs design documents as support for the discussion and >> regular meetings (based on >> https://iceberg.apache.org/contribute/?h=proposal#apache-iceberg-improvement-proposals) >> until doc is finalized, and then it'd move to a PR change, but it seems not >> to be the case for the FGAC proposal. >> >> Is the process still accurate? If so, can we move back the conversations >> related to the FGAC proposal to the mailing list and use the doc as >> support? (and if we can’t move back, can someone explain why this doesn’t >> seem to follow the normal process, what we should have done differently?) >> >> Laurent >> >> On Tue, Sep 16, 2025 at 7:26 AM Robert Stupp <sn...@snazy.de> wrote: >> >>> Hi all, >>> >>> I have updated the proposal document and added a new tab to the Google >>> doc [1]. The textual changes were too much to keep it "inline" and >>> leverage gDoc versioning, so I opted to create a "v2" tab in the same >>> doc. >>> >>> Once the community agrees on the proposal "v2", we can go ahead and >>> discuss the next steps. >>> >>> Best, >>> Robert >>> >>> >>> >>> [1] >>> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38 >>> >>> On Fri, Sep 12, 2025 at 4:38 PM Robert Stupp <sn...@snazy.de> wrote: >>> > >>> > Hi all, >>> > >>> > Regarding the ongoing discussion about the Iceberg REST FGAC OpenAPI >>> > proposal, we want to confirm our alignment with the community's >>> > direction. >>> > >>> > The "Iceberg Expressions + Iceberg UDFs way" as the sole mechanism for >>> > retrieving protection instructions is a good option in my opinion. >>> > This approach aligns with the core principle of separating policy >>> > definitions from protection instructions. >>> > >>> > As the initial proposal highlighted, as good practice / implementation >>> > advice, catalogs should provide user and query-specific protection >>> > instructions. >>> > >>> > We believe this approach effectively addresses the need for >>> > interoperability and the strict separation between how policies are >>> > defined and how they are enforced at query execution time. >>> > >>> > I will update the OpenAPI proposal by next week for broader review. >>> > >>> > Best regards, >>> > Robert >>> > >>> > On Thu, Aug 21, 2025 at 1:22 AM Ryan Blue <rdb...@gmail.com> wrote: >>> > > >>> > > Thanks, Prashant! I agree, it's good to have a starting point we can >>> iterate from. >>> > > >>> > > On Wed, Aug 20, 2025 at 3:25 PM Yufei Gu <flyrain...@gmail.com> >>> wrote: >>> > >> >>> > >> Thanks Prashant for filing the PR. This is a solid start point for >>> FGAC! >>> > >> >>> > >> The spec changes are straightforward and leave plenty of room for >>> future extensibility. While the current Transform [1] isn’t quite >>> sufficient for column masking, enhancements to Transform along with the >>> upcoming UDF proposal should address that gap. For instance, we could >>> introduce a "UDFTerm" once the UDF spec is finalized. >>> > >> >>> > >> [1] >>> https://github.com/polaris-catalog/polaris/blob/8d4cacb27c2a29353091b149e733b4fa7ce4ac53/spec/iceberg-rest-catalog-open-api.yaml#L2317-L2317 >>> > >> >>> > >> Yufei >>> > >> >>> > >> >>> > >> On Wed, Aug 20, 2025 at 2:59 PM Prashant Singh < >>> prashant010...@gmail.com> wrote: >>> > >>> >>> > >>> Robert, >>> > >>> >>> > >>> > It might not change this particular proposal >>> > >>> > having support for UDFs in Iceberg Expressions, which is a >>> prerequisite for this particular proposal >>> > >>> >>> > >>> Unfortunately, that's not my understanding of what community >>> consensus was based on community sync (here) >>> > >>> I have put out a SPEC change proposal based on my understanding of >>> what community consensus is, as PR-13879, I would request other community >>> members to chime here as well. >>> > >>> >>> > >>> As far as client server interaction with UDFs are concerned, based >>> on my understanding of discussion so far in UDF syncs, we want to send all >>> the versions / dialects that udf has from server to the client and the >>> client >>> > >>> picks the one which it wants, not the other way around. It is >>> similar to what we have for Iceberg View already, this should handle both >>> your questions. That being said if you have follow-up questions there is a >>> dedicated Sync for UDF discussions. >>> > >>> >>> > >>> Thank you, >>> > >>> Prashant Singh >>> > >>> >>> > >>> On Wed, Aug 20, 2025 at 2:48 AM Robert Stupp <sn...@snazy.de> >>> wrote: >>> > >>>> >>> > >>>> Prashant, >>> > >>>> >>> > >>>> be ensured that we are still and actively working into this. We >>> are >>> > >>>> looking at this from multiple points of specific views but most >>> > >>>> importantly from a high level view: how all the things work >>> seamlessly >>> > >>>> together. We are really looking forward to providing an update as >>> soon >>> > >>>> as we are confident that the whole approach works. It might not >>> change >>> > >>>> this particular proposal, but it is very important for a proper >>> design >>> > >>>> to think about upstream and downstream impacts. >>> > >>>> >>> > >>>> In the meantime, as it's orthogonal to this particular effort, it >>> > >>>> would be great to get the ball rolling on having support for UDFs >>> in >>> > >>>> Iceberg Expressions, which is a prerequisite for this particular >>> > >>>> proposal. We would be happy, if you could help getting that into >>> > >>>> Apache Iceberg. >>> > >>>> >>> > >>>> As we are speaking about Iceberg UDFs, and implicitly agreed that >>> > >>>> those become a prerequisite for this proposal, it would be very >>> > >>>> beneficial for clients (query engines) being able to leverage the >>> > >>>> following, which is where your help would be appreciated: >>> > >>>> 1. Allows clients to request an Iceberg UDF only in the current >>> > >>>> version, and only the representation (think: SQL dialect) the >>> client >>> > >>>> needs. >>> > >>>> 2. Allow clients to request multiple Iceberg UDF representations >>> in >>> > >>>> one request. Optionally only returning the currentversion and >>> > >>>> optionally restricted to the representation the client needs. >>> > >>>> >>> > >>>> Robert >>> > >>>> >>> > >>>> On Mon, Aug 18, 2025 at 6:05 PM Prashant Singh < >>> prashant010...@gmail.com> wrote: >>> > >>>> > >>> > >>>> > Hi all, >>> > >>>> > >>> > >>>> > It was really nice to go through the proposal in the Iceberg >>> catalog community sync (7/30)! >>> > >>>> > >>> > >>>> > As next steps, I was expecting the doc to be updated according >>> to the community feedback, specifically: >>> > >>>> > >>> > >>>> > Using loadTable API to return the evaluated policies result. >>> > >>>> > >>> > >>>> > Aligning on Iceberg expressions as the path forward for row >>> filters, with UDFs to handle dialect-specific logic. >>> > >>>> > >>> > >>>> > I noticed the doc hasn’t yet been updated with this feedback. I >>> completely understand the authors might be busy, so I’d like to offer help >>> to push this forward. >>> > >>>> > >>> > >>>> > We’ve discussed this topic in depth across several syncs, so >>> perhaps we could even move ahead by opening a spec PR to IRC and continue >>> iterating there. I’m happy to help put that out if it would help keep >>> things moving. >>> > >>>> > >>> > >>>> > Downstream projects such as Apache Polaris are eagerly waiting >>> for this feature, so it would be great to make progress here! >>> > >>>> > >>> > >>>> > Thanks, >>> > >>>> > Prashant Singh >>> > >>>> > >>> > >>>> > >>> > >>>> > On Wed, Jul 23, 2025 at 7:45 AM Robert Stupp <sn...@snazy.de> >>> wrote: >>> > >>>> >> >>> > >>>> >> Hi all, >>> > >>>> >> >>> > >>>> >> Following up on the “Iceberg REST FGAC proposal” discussion >>> [1], we >>> > >>>> >> are happy to share the more detailed proposal [2] to extend >>> the Apache >>> > >>>> >> Iceberg REST specification to include a new API for retrieving >>> > >>>> >> fine-grained access control (FGAC) "protection instructions" >>> > >>>> >> (row-level filters and column transformations) from an Iceberg >>> REST >>> > >>>> >> Catalog. >>> > >>>> >> >>> > >>>> >> The aim is to standardize how query engines obtain these >>> instructions >>> > >>>> >> based on user identity, simplifying data protection >>> enforcement. >>> > >>>> >> >>> > >>>> >> The proposal focuses solely on the new Iceberg REST API >>> endpoint to >>> > >>>> >> retrieve protection instructions, intentionally omitting >>> catalog >>> > >>>> >> specific policy management APIs. >>> > >>>> >> >>> > >>>> >> Having a truly interoperable way to represent the protection >>> > >>>> >> instructions for both row filters and column transformations >>> is a huge >>> > >>>> >> benefit. This is why the support for Iceberg expressions is >>> marked as >>> > >>>> >> mandatory in the proposal. We think that it is a fair option >>> to allow >>> > >>>> >> people to use SQL expressions, not required by the proposal, to >>> > >>>> >> satisfy their needs, assuming they are okay to accept that not >>> all >>> > >>>> >> catalogs or engines support SQL expressions or not all SQL >>> > >>>> >> conformance/dialects. >>> > >>>> >> >>> > >>>> >> Thanks to all of those who have helped review & contribute - JB >>> > >>>> >> Onofre, Prashant Singh, Russell Spitzer, Roy Hansson, & Kevin >>> Liu. We >>> > >>>> >> are excited about the community support! >>> > >>>> >> >>> > >>>> >> Cheers, >>> > >>>> >> Robert, Laurent, Alex, Dmitri >>> > >>>> >> >>> > >>>> >> [1] >>> https://lists.apache.org/thread/nfw1t0glfdfj1hwmzzzzwwyrfnq44yr5 >>> > >>>> >> [2] >>> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM >>> >>
