If this is okay, I would like to keep the beta a bit longer until we see some actual adoption cross-engines, and we address some of the points we discussed previously like the lack of schema for example so we can use the feedback to improve on it and avoid having to release a v2 (or hold off for a long time to refactor).
I'd also like to amend the common table API proposal we discussed last october and hopefully refactor it as an evolution of the existing Generic Table API since it focused amongst other things on the metadata issue. Please let me know if you are agreeable to this. Laurent On Wed, Nov 19, 2025 at 11:03 AM yun zou <[email protected]> wrote: > Hi Dmitri, > > Thanks! > > I have put up the PR: https://github.com/apache/polaris/pull/3096, please > help take a look! > > Will comment on the thread also! > > Best Regards, > Yun > > On Wed, Nov 19, 2025 at 7:00 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Yun, > > > > I am personally ok with the approach you propose. If you would like to > > remove the beta label in 1.3.0, please open a PR to unblock the release > as > > discussed in [1]. If you prefer to remove the label later, please comment > > on [1]. > > > > [1] https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 > > > > Thanks, > > Dmitri. > > > > On Tue, Nov 18, 2025 at 9:47 PM yun zou <[email protected]> > > wrote: > > > > > Hi Dmitri, > > > > > > Thanks for the clarification! > > > > > > -- if we happen to need a major Generic Tables API change after > removing > > > the "beta" label, I believe we'd have to make a v2 of that API > > > > > > Yes, if there are changes that require altering the high-level > semantics > > of > > > the API or modifying existing fields, we would need to move to a V2. > > > > > > However, since the Generic Table API currently defines only very basic > > > fields, I believe the use cases described in the Table Source proposal > > can > > > be addressed by extending the existing APIs. If it eventually becomes > > clear > > > that a major change is required—such as a semantic-level shift—then we > > can > > > transition to V2. The goal is to evolve and build on the current APIs > > > wherever possible. > > > > > > Best Regards, > > > Yun > > > > > > > > > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Yun, > > > > > > > > I personally do not think the "beta" label is related to our > commitment > > > (or > > > > lack of it) in maintaining the API. IMHO, it means that the API is > > likely > > > > to undergo major changes. > > > > > > > > I did not and do not really oppose removing the "beta" label from the > > > > Generic Tables API :) > > > > > > > > My suggestion in keeping it a bit longer was purely to allow more > time > > > > for finding common use cases with the Table Sources proposal and > > possibly > > > > unifying some workflows. > > > > > > > > Having thought about that from that perspective some more, I actually > > > agree > > > > that the Generic Tables API is _not_ likely to undergo major changes. > > If > > > > synergies with Table Sources can be found, most of the changes are > > > probably > > > > going to happen in clients that currently use the Generic Tables API, > > the > > > > API itself is probably going to remain stable. > > > > > > > > That said, just for the sake of clarity, if we happen to need a major > > > > Generic Tables API change after removing the "beta" label, I believe > > we'd > > > > have to make a v2 of that API (which is ok from my POV per our > > evolution > > > > guidelines [1]). > > > > > > > > [1] https://polaris.apache.org/in-dev/unreleased/evolution/ > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou <[email protected]> > > > > wrote: > > > > > > > > > Hi Dmitri, > > > > > > > > > > Thanks a lot for the feedback! > > > > > > > > > > I definitely agree that the current generic table support still has > > > > > limitations and that more work is needed. However, removing the > > “beta” > > > > > label doesn’t mean the feature is finalized. Instead, it signals to > > > users > > > > > that we are committed to maintaining and improving it over time. > > > > Therefore, > > > > > I believe this will not block any future enhancements including > > > extending > > > > > it to address the use cases described in the Table Source proposal. > > > > > > > > > > As Adam pointed out, we already have users trying it out and > > requesting > > > > > improvements. I believe this is a good time to remove the “beta” > > label > > > to > > > > > encourage broader adoption and welcome new contributors who can > help > > > > > advance the feature. > > > > > > > > > > Best Regards, > > > > > Yun > > > > > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < > > > > > [email protected]> wrote: > > > > > > > > > > > I'm in favor of removing the "beta" label because it is starting > to > > > be > > > > > used > > > > > > by users. From the Slack feedback we have seen, there are users > who > > > > > > have started to try out this feature. While they are still > running > > > into > > > > > > issues such as not having a Spark 4 plugin [1] and not having > > > > credential > > > > > > vending [2], there is real user usage indicating that folks find > > this > > > > > > beneficial. > > > > > > > > > > > > Now, I do agree with Dmitri that there are known limitations we > > need > > > to > > > > > > handle that are impeding users. The two referenced examples above > > are > > > > > > obvious ones that need fixes to get wider adoption. I have been > > > working > > > > > > with Yun & others to be able to solve some of these issues (like > > this > > > > PR > > > > > > [3]), but I agree that there may need to be more significant > > changes > > > > > > including a REST API change. > > > > > > > > > > > > From my understanding, during our last conversation in the > > community > > > > > about > > > > > > the "Table Sources" proposal [4], we decided we were going to > > analyze > > > > the > > > > > > Parquet file use case and see if Generic Tables can support this > > use > > > > case > > > > > > or whether we need to bring some ideas from the Table Sources > > > proposal > > > > > into > > > > > > Generic Tables. I believe that this approach is still valid and > > will > > > > > > benefit our adopting users by identifying any enhancements with > > these > > > > new > > > > > > use cases. > > > > > > > > > > > > What do y'all think? > > > > > > > > > > > > [1] - https://github.com/apache/polaris/issues/3021 > > > > > > [2] - https://github.com/apache/polaris/issues/3020 > > > > > > [3] - https://github.com/apache/polaris/pull/3026 > > > > > > [4] - > > > https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 > > > > > > > > > > > > Go community, > > > > > > > > > > > > Adam > > > > > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri Bourlatchkov < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > We should indeed review the status of the Generic Tables API, > so > > > > thanks > > > > > > for > > > > > > > starting this discussion! > > > > > > > > > > > > > > From my POV the key question is: how do we intend to proceed > with > > > > known > > > > > > > limitations [1]? > > > > > > > > > > > > > > I believe the Table Sources proposal [2] addresses some (if not > > > all) > > > > of > > > > > > > those limitations. It is certainly suitable for the same > > > applications > > > > > > that > > > > > > > currently go through the Generic Tables API. > > > > > > > > > > > > > > I believe it would be wise to allow the Table Sources proposal > to > > > > > develop > > > > > > > further to see if there are any synergies that can be leveraged > > > with > > > > > > > respect to Generic Tables. If we removed the "beta" label from > > the > > > > > > Generic > > > > > > > Tables API now, it would make it harder for users to benefit > from > > > > those > > > > > > > synergies later (due to virtually freezing the "spec"). > > > > > > > > > > > > > > At it stands now, the Generic Tables API is supported by > Polaris, > > > so > > > > > > > existing clients can continue operating normally. > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 > > > > > > > > > > > > > > [2] > > > https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > Generic Table has been available since Polaris 1.0 and has > > > > attracted > > > > > > > > interest from several users. We also have ongoing improvement > > and > > > > > > > extension > > > > > > > > work in progress, including Hudi support, Parquet support, > and > > > > > > credential > > > > > > > > vending. > > > > > > > > > > > > > > > > Given this progress, I believe it’s a good time to remove the > > > > “beta” > > > > > > > label. > > > > > > > > If there are no objections, we will remove the “beta” label > > from > > > > the > > > > > > > > Generic Table in Polaris 1.3. > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
