Hi Dmitri, Thanks!
I have put up the PR: https://github.com/apache/polaris/pull/3096, please help take a look! Will comment on the thread also! Best Regards, Yun On Wed, Nov 19, 2025 at 7:00 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yun, > > I am personally ok with the approach you propose. If you would like to > remove the beta label in 1.3.0, please open a PR to unblock the release as > discussed in [1]. If you prefer to remove the label later, please comment > on [1]. > > [1] https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 > > Thanks, > Dmitri. > > On Tue, Nov 18, 2025 at 9:47 PM yun zou <[email protected]> > wrote: > > > Hi Dmitri, > > > > Thanks for the clarification! > > > > -- if we happen to need a major Generic Tables API change after removing > > the "beta" label, I believe we'd have to make a v2 of that API > > > > Yes, if there are changes that require altering the high-level semantics > of > > the API or modifying existing fields, we would need to move to a V2. > > > > However, since the Generic Table API currently defines only very basic > > fields, I believe the use cases described in the Table Source proposal > can > > be addressed by extending the existing APIs. If it eventually becomes > clear > > that a major change is required—such as a semantic-level shift—then we > can > > transition to V2. The goal is to evolve and build on the current APIs > > wherever possible. > > > > Best Regards, > > Yun > > > > > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Yun, > > > > > > I personally do not think the "beta" label is related to our commitment > > (or > > > lack of it) in maintaining the API. IMHO, it means that the API is > likely > > > to undergo major changes. > > > > > > I did not and do not really oppose removing the "beta" label from the > > > Generic Tables API :) > > > > > > My suggestion in keeping it a bit longer was purely to allow more time > > > for finding common use cases with the Table Sources proposal and > possibly > > > unifying some workflows. > > > > > > Having thought about that from that perspective some more, I actually > > agree > > > that the Generic Tables API is _not_ likely to undergo major changes. > If > > > synergies with Table Sources can be found, most of the changes are > > probably > > > going to happen in clients that currently use the Generic Tables API, > the > > > API itself is probably going to remain stable. > > > > > > That said, just for the sake of clarity, if we happen to need a major > > > Generic Tables API change after removing the "beta" label, I believe > we'd > > > have to make a v2 of that API (which is ok from my POV per our > evolution > > > guidelines [1]). > > > > > > [1] https://polaris.apache.org/in-dev/unreleased/evolution/ > > > > > > Cheers, > > > Dmitri. > > > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou <[email protected]> > > > wrote: > > > > > > > Hi Dmitri, > > > > > > > > Thanks a lot for the feedback! > > > > > > > > I definitely agree that the current generic table support still has > > > > limitations and that more work is needed. However, removing the > “beta” > > > > label doesn’t mean the feature is finalized. Instead, it signals to > > users > > > > that we are committed to maintaining and improving it over time. > > > Therefore, > > > > I believe this will not block any future enhancements including > > extending > > > > it to address the use cases described in the Table Source proposal. > > > > > > > > As Adam pointed out, we already have users trying it out and > requesting > > > > improvements. I believe this is a good time to remove the “beta” > label > > to > > > > encourage broader adoption and welcome new contributors who can help > > > > advance the feature. > > > > > > > > Best Regards, > > > > Yun > > > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < > > > > [email protected]> wrote: > > > > > > > > > I'm in favor of removing the "beta" label because it is starting to > > be > > > > used > > > > > by users. From the Slack feedback we have seen, there are users who > > > > > have started to try out this feature. While they are still running > > into > > > > > issues such as not having a Spark 4 plugin [1] and not having > > > credential > > > > > vending [2], there is real user usage indicating that folks find > this > > > > > beneficial. > > > > > > > > > > Now, I do agree with Dmitri that there are known limitations we > need > > to > > > > > handle that are impeding users. The two referenced examples above > are > > > > > obvious ones that need fixes to get wider adoption. I have been > > working > > > > > with Yun & others to be able to solve some of these issues (like > this > > > PR > > > > > [3]), but I agree that there may need to be more significant > changes > > > > > including a REST API change. > > > > > > > > > > From my understanding, during our last conversation in the > community > > > > about > > > > > the "Table Sources" proposal [4], we decided we were going to > analyze > > > the > > > > > Parquet file use case and see if Generic Tables can support this > use > > > case > > > > > or whether we need to bring some ideas from the Table Sources > > proposal > > > > into > > > > > Generic Tables. I believe that this approach is still valid and > will > > > > > benefit our adopting users by identifying any enhancements with > these > > > new > > > > > use cases. > > > > > > > > > > What do y'all think? > > > > > > > > > > [1] - https://github.com/apache/polaris/issues/3021 > > > > > [2] - https://github.com/apache/polaris/issues/3020 > > > > > [3] - https://github.com/apache/polaris/pull/3026 > > > > > [4] - > > https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 > > > > > > > > > > Go community, > > > > > > > > > > Adam > > > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri Bourlatchkov < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > We should indeed review the status of the Generic Tables API, so > > > thanks > > > > > for > > > > > > starting this discussion! > > > > > > > > > > > > From my POV the key question is: how do we intend to proceed with > > > known > > > > > > limitations [1]? > > > > > > > > > > > > I believe the Table Sources proposal [2] addresses some (if not > > all) > > > of > > > > > > those limitations. It is certainly suitable for the same > > applications > > > > > that > > > > > > currently go through the Generic Tables API. > > > > > > > > > > > > I believe it would be wise to allow the Table Sources proposal to > > > > develop > > > > > > further to see if there are any synergies that can be leveraged > > with > > > > > > respect to Generic Tables. If we removed the "beta" label from > the > > > > > Generic > > > > > > Tables API now, it would make it harder for users to benefit from > > > those > > > > > > synergies later (due to virtually freezing the "spec"). > > > > > > > > > > > > At it stands now, the Generic Tables API is supported by Polaris, > > so > > > > > > existing clients can continue operating normally. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 > > > > > > > > > > > > [2] > > https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Generic Table has been available since Polaris 1.0 and has > > > attracted > > > > > > > interest from several users. We also have ongoing improvement > and > > > > > > extension > > > > > > > work in progress, including Hudi support, Parquet support, and > > > > > credential > > > > > > > vending. > > > > > > > > > > > > > > Given this progress, I believe it’s a good time to remove the > > > “beta” > > > > > > label. > > > > > > > If there are no objections, we will remove the “beta” label > from > > > the > > > > > > > Generic Table in Polaris 1.3. > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > >
