One way that you might be able to reach some users is to issue a warning in the code for SqlTransform.expand(), with the deprecation timeline. Maybe this is under "etc" but I wanted to mention it because the other items were all documentation.
Kenn On Mon, Mar 31, 2025 at 1:34 PM Yi Hu via dev <dev@beam.apache.org> wrote: > Thanks for inputs! From discussion we have agreed on moving forward > > Next step: > > - Draft PR to note the deprecation status in Documentation, including > CHANGES, Javadoc, etc > > - The earliest release to stop publishing ZetaSQL artifacts is pushed > further, not earlier than 1 full quarter before the next Release (2.65.0 + > 3 months would be 2.68.0), and only when it causes issues on maintenance. > > Thanks, > > Yi > > > On Wed, Mar 26, 2025 at 12:18 PM Kenneth Knowles <k...@apache.org> wrote: > >> +1 to this deprecation. Thanks for putting together a clear summary. >> >> FWIW it also has significantly worse performance than Calcite SQL >> dialect, since it calls out to a ZetaSQL subprocess for most calculations, >> and that is less optimized than Beam's Fn API. >> >> Kenn >> >> On Tue, Mar 25, 2025 at 4:18 PM Robert Bradshaw via user < >> u...@beam.apache.org> wrote: >> >>> I'm in favor of deprecating this and cleaning it up, but it depends on >>> usage. I suspect it is low (or possibly non-existent, especially as there's >>> little upside to moving away from the default). I cc'd user@ just in >>> case anyone wants to chime in there. This may be a good thing to add to our >>> release notes as well (perhaps we can get it in the one that's just about >>> to go out). >>> >>> Unless there is strong, justified pushback, I'd get the deprecation >>> status (e.g. on the javadocs, website) right away. For actual removal, I >>> agree with the idea of waiting until it actually causes issues or we move >>> to the next major beam release, though I might push back at 2.66 being a >>> bit too quick even if the first condition is hit before then and might give >>> people at least a quarter's notice. >>> >>> - Robert >>> >>> >>> On Mon, Mar 24, 2025 at 2:27 PM Yi Hu via dev <dev@beam.apache.org> >>> wrote: >>> >>>> Hi everyone, >>>> >>>> I would like to bring up discussion for deprecating Beam SQL's ZetaSQL >>>> component [1]. >>>> Beam SQL currently serves with two SQL dialects (i) Apache Calcite and >>>> (ii) ZetaSQL dialects, see documentation [2] due to the following reasons >>>> >>>> - Developments in Beam for ZetaSQL dialect effectively stalled since >>>> early 2022 (See change history [3]) >>>> >>>> - Despite incomplete support status, there is no new bug / feature >>>> request opened ever since we migrated to use GitHub Issue, suggesting >>>> minimal adoption [4] >>>> >>>> - We still need to keep zetasql up-to-date if its dependency conflicts >>>> with other google dependencies, as a result ZetaSQL component introduces >>>> maintenance burden when upgrading GCP-BOM (e.g. [5]). >>>> >>>> - One of the main reason that using ZetaSQL dialect, per [2], was >>>> because >>>> >>>> > ZetaSQL is more compatible with BigQuery, so it’s especially useful >>>> in pipelines that write to or read from BigQuery tables. >>>> >>>> As of today, as GCP BigQuery now supports using GoogleSQL >>>> (open-sourced as ZetaSQL) querying data that's stored outside of BigQuery >>>> via BigQuery Connections API / Federated query [6, 7]. This largely >>>> provides an alternative for using Beam's ZetaSQL interacting with BigQuery. >>>> >>>> For these reasons, I propose initiating the process of deprecating >>>> Beam SQL's ZetaSQL component. There are two decisions needed to be made: >>>> >>>> Firstly, agree on when to document the deprecated status for ZetaSQL >>>> component in javadoc, beam website, currently I recommend do it in the >>>> release that currently HEAD belongs, that is Beam 2.65.0 (cut April 30, >>>> 2025) >>>> >>>> Secondly, stop publishing ZetaSQL artifacts. This is a breaking change, >>>> and I think we can leave the deprecated status as is until the following >>>> situation emerges, whichever comes first, and no earlier than Beam 2.66.0 >>>> (cut Jun 11, 2025) >>>> >>>> - Continued support for ZetaSQL component involving significant >>>> burdens, like conflict with other Beam dependencies, supported Java >>>> versions, etc, or >>>> - When Beam moved to the next release major release (3) >>>> >>>> Thanks for your attention, and any input welcomed! >>>> >>>> Regards, >>>> Yi >>>> >>>> [1] >>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/zetasql >>>> [2] https://beam.apache.org/documentation/dsls/sql/overview/ >>>> [3] >>>> https://github.com/benEng/beam/commits/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java >>>> [4] >>>> https://github.com/apache/beam/issues?q=is%3Aissue%20%20label%3Azetasql%20 >>>> [5] https://github.com/apache/beam/pull/32902 >>>> [6] https://cloud.google.com/bigquery/docs/connections-api-intro >>>> [7] https://cloud.google.com/bigquery/docs/federated-queries-intro >>>> >>>> -- >>>> >>>> Yi Hu, (he/him/his) >>>> >>>> Software Engineer >>>> >>>> >>>>
