Thanks, Ajantha. I'm skeptical about whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily deals with things that are centralized, like tables of data. While it would be great to have a common set of functions across engines, I don't see how that is practical when those engines are implemented so differently. Plugging in code -- and especially custom user-supplied code -- seems inherently specialized to me and should be part of the engines' design.
I guess we'll know more when you post the proposal, but I think this would be a very difficult area to tackle across engines, languages, and memory models without having a huge performance penalty. Ryan On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <ajanthab...@gmail.com> wrote: > Hi Everyone, > > This is a discussion to gauge the community interest in storing the > Versioned SQL UDFs in Iceberg. > We want to propose the spec addition for storing the versioned UDFs in > Iceberg (inspired by view spec). > > These UDFs can operate similarly to views in that they are associated with > tables, but they can accept arguments and produce return values, or even > function as inline expressions. > Many Query engines like Dremio, Trino, Snowflake, Databricks Spark > supports SQL UDFs at catalog level [1]. > But storing them in Iceberg can enable > - Versioning of these UDFs. > - Interoperability between the engines. Potentially engines can understand > the UDFs written by other engines (with the translate layer). > > We believe that integrating this feature into Iceberg would be a valuable > addition, and we're eager to collaborate with the community to develop a > UDF specification. > Stephen <stephen....@dremio.com> has already begun drafting a > specification to propose to the community. > > Let us know your thoughts on this. > > [1] > Dremio - > https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function > Trino - https://trino.io/docs/current/sql/create-function.html > Snowflake - > https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions > Databricks - > https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html > > - Ajantha > -- Ryan Blue Tabular
