Hi JB, Sorry about that!
I went ahead with the merge since Laurent mentioned he was ok to move forward as long as the rest of the community had reached consensus, and it seemed that most people were aligned. I’m happy to revert it if you prefer. Regarding the limitations: the current documentation already includes a clear section outlining the feature’s limitations [1]. If there are additional points you think should be covered, just let me know and I can definitely add them in. [1]: https://polaris.apache.org/releases/1.2.0/generic-table/ Best Regards, Yun On Mon, Nov 24, 2025 at 1:13 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi All, > > I noticed that PR #3096 was merged before this discussion reached a > clear consensus. Moving forward, I think it would be beneficial to > ensure we have community consensus before merging, especially without > an update on the thread. > > Regarding Laurent's concerns, I am fine with removing the "beta" flag > for the Generic Table API, however, I suggest updating the Generic > Table documentation to clearly state its current limitations and > proposed future evolutions. > > Since documenting the limitations is not a blocker for the upcoming > release, I propose we proceed with the 1.3.0-incubating release and > address the documentation update immediately following the release. > > Regards, > JB > > On Sat, Nov 22, 2025 at 3:17 PM Jean-Baptiste Onofré <[email protected]> > wrote: > > > > Sorry I meant 1.3.0-incubating and 1.4.0-incubating in terms of releases. > > > > Le sam. 22 nov. 2025 à 11:27, Jean-Baptiste Onofré <[email protected]> a > écrit : > >> > >> Hi All, > >> > >> I understand Laurent’s points and agree with the sentiment regarding > >> the API’s future evolution. The current Generic Table API, despite its > >> limitations (like the Delta and CSV support demonstrates), is > >> functional. > >> > >> As a community, we need to achieve consensus on this topic. Consensus > >> does not require unanimous agreement on every detail but means the > >> project has reached a decision or compromise that everyone can accept. > >> Specifically, using lazy consensus means we can proceed unless there > >> is a legitimate objection accompanied by an alternative approach for > >> discussion. > >> > >> To move toward consensus and incorporate Laurent's concerns, I propose > >> the following compromise: > >> > >> 1. Keep the "beta" flag for the upcoming 1.2.0-incubating release. > >> 2. Re-evaluate and aim to remove the "beta" label for the > >> 1.3.0-incubating release. > >> > >> For the 1.3.0-incubating release to proceed without the "beta" label, > >> I suggest we clearly document the existing limitations of the Generic > >> Table API and reference the relevant discussions/issues concerning its > >> evolution (specifically the issues for schema support and credential > >> vending, as well as the Common Table API proposal). > >> > >> Does this compromise seem reasonable to everyone? > >> > >> If this approach does not gain approval, we can initiate a formal > >> vote, though I believe that may be unnecessary here. > >> > >> Regards, > >> JB > >> > >> On Fri, Nov 21, 2025 at 10:17 PM yun zou <[email protected]> > wrote: > >> > > >> > Hi Laurent, > >> > > >> > Thanks a lot for the input! I fully agree that there’s still plenty > of room > >> > for improvement before this feature reaches a perfect state. > Meanwhile, > >> > since the feature is already attracting interest, I think it would be > >> > valuable to remove the “beta” label so we can draw wider attention and > >> > collect more feedback to help it evolve in the right direction. > >> > > >> > As JB pointed out, this won’t block the evolution of the API. If we > need to > >> > introduce a V2 in the future, I think that’s perfectly fine—it would > show > >> > that the Polaris community is committed to continuously improving and > >> > supporting non-Iceberg standards. > >> > > >> > Since we’ve mostly aligned on removing the beta label, how about we > proceed > >> > with that? > >> > > >> > > >> > Best Regards, > >> > Yun > >> > > >> > On Fri, Nov 21, 2025 at 8:58 AM Laurent Goujon <[email protected]> > wrote: > >> > > >> > > Sorry didn't want to be difficult but I was just hoping we could > address > >> > > some of the possible issuers we flagged when we decided to add the > beta > >> > > status to the api in the first place. I agree there's some great > interest > >> > > for it (we have been chatting about it also at the south bay meet > up this > >> > > week) but I don't know about adoption? But I also respect the > willingness > >> > > to move fast even if I think that keeping a beta flag for a long > period of > >> > > time is not necessarily a bad thing (maybe in the future we can > separate > >> > > experimental where things may be really removed or totally reworked > from > >> > > beta where the core structure is in place and we will try to > minimize > >> > > breaking changes) > >> > > > >> > > Ultimately it was just a simple request, if the consensus is to > remove the > >> > > beta flag, then please ignore my previous email and proceed. > >> > > > >> > > Laurent > >> > > > >> > > On Fri, Nov 21, 2025 at 6:08 AM Jean-Baptiste Onofré < > [email protected]> > >> > > wrote: > >> > > > >> > > > Hi All, > >> > > > > >> > > > I think it is important to distinguish between the "beta" status > of > >> > > > the Generic Table API and its future evolution. > >> > > > > >> > > > Strictly speaking, I believe we should remove the "beta" label > now, as > >> > > > the API already has several implementations and is seeing > real-world > >> > > > usage. > >> > > > > >> > > > While we all agree that the Generic Table API will continue to > evolve > >> > > > (e.g., credential vending, common table API), I anticipate that > these > >> > > > changes will primarily be "additions" rather than breaking > changes to > >> > > > the existing specification (which is pretty thin honestly). For > >> > > > example, adding schema support should be backward compatible. > >> > > > > >> > > > Personally, I would rather avoid using the "beta" flag here. > Features > >> > > > in open source projects are inherently evolving, often rapidly. > >> > > > > >> > > > Just my two cents. > >> > > > > >> > > > Regards, > >> > > > JB > >> > > > > >> > > > On Fri, Nov 21, 2025 at 4:46 AM Laurent Goujon < > [email protected]> > >> > > wrote: > >> > > > > > >> > > > > If this is okay, I would like to keep the beta a bit longer > until we > >> > > see > >> > > > > some actual adoption cross-engines, and we address some of the > points > >> > > we > >> > > > > discussed previously like the lack of schema for example so we > can use > >> > > > the > >> > > > > feedback to improve on it and avoid having to release a v2 (or > hold off > >> > > > for > >> > > > > a long time to refactor). > >> > > > > > >> > > > > I'd also like to amend the common table API proposal we > discussed last > >> > > > > october and hopefully refactor it as an evolution of the > existing > >> > > Generic > >> > > > > Table API since it focused amongst other things on the metadata > issue. > >> > > > > > >> > > > > Please let me know if you are agreeable to this. > >> > > > > > >> > > > > Laurent > >> > > > > > >> > > > > On Wed, Nov 19, 2025 at 11:03 AM yun zou < > [email protected]> > >> > > > wrote: > >> > > > > > >> > > > > > Hi Dmitri, > >> > > > > > > >> > > > > > Thanks! > >> > > > > > > >> > > > > > I have put up the PR: > https://github.com/apache/polaris/pull/3096, > >> > > > please > >> > > > > > help take a look! > >> > > > > > > >> > > > > > Will comment on the thread also! > >> > > > > > > >> > > > > > Best Regards, > >> > > > > > Yun > >> > > > > > > >> > > > > > On Wed, Nov 19, 2025 at 7:00 AM Dmitri Bourlatchkov < > >> > > [email protected]> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hi Yun, > >> > > > > > > > >> > > > > > > I am personally ok with the approach you propose. If you > would like > >> > > > to > >> > > > > > > remove the beta label in 1.3.0, please open a PR to unblock > the > >> > > > release > >> > > > > > as > >> > > > > > > discussed in [1]. If you prefer to remove the label later, > please > >> > > > comment > >> > > > > > > on [1]. > >> > > > > > > > >> > > > > > > [1] > >> > > https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > Dmitri. > >> > > > > > > > >> > > > > > > On Tue, Nov 18, 2025 at 9:47 PM yun zou < > >> > > [email protected]> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi Dmitri, > >> > > > > > > > > >> > > > > > > > Thanks for the clarification! > >> > > > > > > > > >> > > > > > > > -- if we happen to need a major Generic Tables API change > after > >> > > > > > removing > >> > > > > > > > the "beta" label, I believe we'd have to make a v2 of > that API > >> > > > > > > > > >> > > > > > > > Yes, if there are changes that require altering the > high-level > >> > > > > > semantics > >> > > > > > > of > >> > > > > > > > the API or modifying existing fields, we would need to > move to a > >> > > > V2. > >> > > > > > > > > >> > > > > > > > However, since the Generic Table API currently defines > only very > >> > > > basic > >> > > > > > > > fields, I believe the use cases described in the Table > Source > >> > > > proposal > >> > > > > > > can > >> > > > > > > > be addressed by extending the existing APIs. If it > eventually > >> > > > becomes > >> > > > > > > clear > >> > > > > > > > that a major change is required—such as a semantic-level > >> > > > shift—then we > >> > > > > > > can > >> > > > > > > > transition to V2. The goal is to evolve and build on the > current > >> > > > APIs > >> > > > > > > > wherever possible. > >> > > > > > > > > >> > > > > > > > Best Regards, > >> > > > > > > > Yun > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov < > >> > > > [email protected]> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hi Yun, > >> > > > > > > > > > >> > > > > > > > > I personally do not think the "beta" label is related > to our > >> > > > > > commitment > >> > > > > > > > (or > >> > > > > > > > > lack of it) in maintaining the API. IMHO, it means that > the API > >> > > > is > >> > > > > > > likely > >> > > > > > > > > to undergo major changes. > >> > > > > > > > > > >> > > > > > > > > I did not and do not really oppose removing the "beta" > label > >> > > > from the > >> > > > > > > > > Generic Tables API :) > >> > > > > > > > > > >> > > > > > > > > My suggestion in keeping it a bit longer was purely to > allow > >> > > more > >> > > > > > time > >> > > > > > > > > for finding common use cases with the Table Sources > proposal > >> > > and > >> > > > > > > possibly > >> > > > > > > > > unifying some workflows. > >> > > > > > > > > > >> > > > > > > > > Having thought about that from that perspective some > more, I > >> > > > actually > >> > > > > > > > agree > >> > > > > > > > > that the Generic Tables API is _not_ likely to undergo > major > >> > > > changes. > >> > > > > > > If > >> > > > > > > > > synergies with Table Sources can be found, most of the > changes > >> > > > are > >> > > > > > > > probably > >> > > > > > > > > going to happen in clients that currently use the > Generic > >> > > Tables > >> > > > API, > >> > > > > > > the > >> > > > > > > > > API itself is probably going to remain stable. > >> > > > > > > > > > >> > > > > > > > > That said, just for the sake of clarity, if we happen > to need a > >> > > > major > >> > > > > > > > > Generic Tables API change after removing the "beta" > label, I > >> > > > believe > >> > > > > > > we'd > >> > > > > > > > > have to make a v2 of that API (which is ok from my POV > per our > >> > > > > > > evolution > >> > > > > > > > > guidelines [1]). > >> > > > > > > > > > >> > > > > > > > > [1] > https://polaris.apache.org/in-dev/unreleased/evolution/ > >> > > > > > > > > > >> > > > > > > > > Cheers, > >> > > > > > > > > Dmitri. > >> > > > > > > > > > >> > > > > > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou < > >> > > > [email protected]> > >> > > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Hi Dmitri, > >> > > > > > > > > > > >> > > > > > > > > > Thanks a lot for the feedback! > >> > > > > > > > > > > >> > > > > > > > > > I definitely agree that the current generic table > support > >> > > > still has > >> > > > > > > > > > limitations and that more work is needed. However, > removing > >> > > the > >> > > > > > > “beta” > >> > > > > > > > > > label doesn’t mean the feature is finalized. Instead, > it > >> > > > signals to > >> > > > > > > > users > >> > > > > > > > > > that we are committed to maintaining and improving it > over > >> > > > time. > >> > > > > > > > > Therefore, > >> > > > > > > > > > I believe this will not block any future enhancements > >> > > including > >> > > > > > > > extending > >> > > > > > > > > > it to address the use cases described in the Table > Source > >> > > > proposal. > >> > > > > > > > > > > >> > > > > > > > > > As Adam pointed out, we already have users trying it > out and > >> > > > > > > requesting > >> > > > > > > > > > improvements. I believe this is a good time to remove > the > >> > > > “beta” > >> > > > > > > label > >> > > > > > > > to > >> > > > > > > > > > encourage broader adoption and welcome new > contributors who > >> > > can > >> > > > > > help > >> > > > > > > > > > advance the feature. > >> > > > > > > > > > > >> > > > > > > > > > Best Regards, > >> > > > > > > > > > Yun > >> > > > > > > > > > > >> > > > > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < > >> > > > > > > > > > [email protected]> wrote: > >> > > > > > > > > > > >> > > > > > > > > > > I'm in favor of removing the "beta" label because > it is > >> > > > starting > >> > > > > > to > >> > > > > > > > be > >> > > > > > > > > > used > >> > > > > > > > > > > by users. From the Slack feedback we have seen, > there are > >> > > > users > >> > > > > > who > >> > > > > > > > > > > have started to try out this feature. While they > are still > >> > > > > > running > >> > > > > > > > into > >> > > > > > > > > > > issues such as not having a Spark 4 plugin [1] and > not > >> > > having > >> > > > > > > > > credential > >> > > > > > > > > > > vending [2], there is real user usage indicating > that folks > >> > > > find > >> > > > > > > this > >> > > > > > > > > > > beneficial. > >> > > > > > > > > > > > >> > > > > > > > > > > Now, I do agree with Dmitri that there are known > >> > > limitations > >> > > > we > >> > > > > > > need > >> > > > > > > > to > >> > > > > > > > > > > handle that are impeding users. The two referenced > examples > >> > > > above > >> > > > > > > are > >> > > > > > > > > > > obvious ones that need fixes to get wider adoption. > I have > >> > > > been > >> > > > > > > > working > >> > > > > > > > > > > with Yun & others to be able to solve some of these > issues > >> > > > (like > >> > > > > > > this > >> > > > > > > > > PR > >> > > > > > > > > > > [3]), but I agree that there may need to be more > >> > > significant > >> > > > > > > changes > >> > > > > > > > > > > including a REST API change. > >> > > > > > > > > > > > >> > > > > > > > > > > From my understanding, during our last conversation > in the > >> > > > > > > community > >> > > > > > > > > > about > >> > > > > > > > > > > the "Table Sources" proposal [4], we decided we > were going > >> > > to > >> > > > > > > analyze > >> > > > > > > > > the > >> > > > > > > > > > > Parquet file use case and see if Generic Tables can > support > >> > > > this > >> > > > > > > use > >> > > > > > > > > case > >> > > > > > > > > > > or whether we need to bring some ideas from the > Table > >> > > Sources > >> > > > > > > > proposal > >> > > > > > > > > > into > >> > > > > > > > > > > Generic Tables. I believe that this approach is > still valid > >> > > > and > >> > > > > > > will > >> > > > > > > > > > > benefit our adopting users by identifying any > enhancements > >> > > > with > >> > > > > > > these > >> > > > > > > > > new > >> > > > > > > > > > > use cases. > >> > > > > > > > > > > > >> > > > > > > > > > > What do y'all think? > >> > > > > > > > > > > > >> > > > > > > > > > > [1] - https://github.com/apache/polaris/issues/3021 > >> > > > > > > > > > > [2] - https://github.com/apache/polaris/issues/3020 > >> > > > > > > > > > > [3] - https://github.com/apache/polaris/pull/3026 > >> > > > > > > > > > > [4] - > >> > > > > > > > > https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 > >> > > > > > > > > > > > >> > > > > > > > > > > Go community, > >> > > > > > > > > > > > >> > > > > > > > > > > Adam > >> > > > > > > > > > > > >> > > > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri > Bourlatchkov < > >> > > > > > > > [email protected] > >> > > > > > > > > > > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > Hi Yun, > >> > > > > > > > > > > > > >> > > > > > > > > > > > We should indeed review the status of the Generic > Tables > >> > > > API, > >> > > > > > so > >> > > > > > > > > thanks > >> > > > > > > > > > > for > >> > > > > > > > > > > > starting this discussion! > >> > > > > > > > > > > > > >> > > > > > > > > > > > From my POV the key question is: how do we intend > to > >> > > > proceed > >> > > > > > with > >> > > > > > > > > known > >> > > > > > > > > > > > limitations [1]? > >> > > > > > > > > > > > > >> > > > > > > > > > > > I believe the Table Sources proposal [2] > addresses some > >> > > > (if not > >> > > > > > > > all) > >> > > > > > > > > of > >> > > > > > > > > > > > those limitations. It is certainly suitable for > the same > >> > > > > > > > applications > >> > > > > > > > > > > that > >> > > > > > > > > > > > currently go through the Generic Tables API. > >> > > > > > > > > > > > > >> > > > > > > > > > > > I believe it would be wise to allow the Table > Sources > >> > > > proposal > >> > > > > > to > >> > > > > > > > > > develop > >> > > > > > > > > > > > further to see if there are any synergies that > can be > >> > > > leveraged > >> > > > > > > > with > >> > > > > > > > > > > > respect to Generic Tables. If we removed the > "beta" label > >> > > > from > >> > > > > > > the > >> > > > > > > > > > > Generic > >> > > > > > > > > > > > Tables API now, it would make it harder for users > to > >> > > > benefit > >> > > > > > from > >> > > > > > > > > those > >> > > > > > > > > > > > synergies later (due to virtually freezing the > "spec"). > >> > > > > > > > > > > > > >> > > > > > > > > > > > At it stands now, the Generic Tables API is > supported by > >> > > > > > Polaris, > >> > > > > > > > so > >> > > > > > > > > > > > existing clients can continue operating normally. > >> > > > > > > > > > > > > >> > > > > > > > > > > > [1] > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > >> > > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 > >> > > > > > > > > > > > > >> > > > > > > > > > > > [2] > >> > > > > > > > > https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > Dmitri. > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < > >> > > > > > > > [email protected] > >> > > > > > > > > > > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Hi All, > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Generic Table has been available since Polaris > 1.0 and > >> > > > has > >> > > > > > > > > attracted > >> > > > > > > > > > > > > interest from several users. We also have > ongoing > >> > > > improvement > >> > > > > > > and > >> > > > > > > > > > > > extension > >> > > > > > > > > > > > > work in progress, including Hudi support, > Parquet > >> > > > support, > >> > > > > > and > >> > > > > > > > > > > credential > >> > > > > > > > > > > > > vending. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Given this progress, I believe it’s a good time > to > >> > > > remove the > >> > > > > > > > > “beta” > >> > > > > > > > > > > > label. > >> > > > > > > > > > > > > If there are no objections, we will remove the > “beta” > >> > > > label > >> > > > > > > from > >> > > > > > > > > the > >> > > > > > > > > > > > > Generic Table in Polaris 1.3. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Best Regards, > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Yun > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > >> > > >
