Hi everyone,

We've only received one review so far (from Benny).

We would appreciate more eyes on this.

- Ajantha

On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <ajanthab...@gmail.com> wrote:

> Hi All,
> Please find the proposal link
> https://github.com/apache/iceberg/issues/10432
>
> Google doc link is attached in the proposal.
> And Thanks Stephen Lin <https://github.com/sxlin> for working on it.
>
> Hope it gives more clarity to take the decisions and how we want to
> implement it.
>
> - Ajantha
>
> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
> wa.moust...@gmail.com> wrote:
>
>> Thanks Jack. I actually meant scalar/aggregate/table user defined
>> functions. Here are some examples of what I meant in (2):
>>
>> Hive GenericUDF:
>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>> Trino user defined functions:
>> https://trino.io/docs/current/develop/functions.html
>> Flink user defined functions:
>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>
>> Probably what you referred to is a variation of (1) where the API is data
>> flow/data pipeline API instead of SQL (e.g., Spark Scala). Yes, that is
>> also possible in the very long run :)
>>
>> Thanks,
>> Walaa.
>>
>>
>>
>>
>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <yezhao...@gmail.com> wrote:
>>
>>> > (2) Custom code written in imperative function according to a
>>> Java/Scala/Python API, etc.
>>>
>>> I think we could still explore some long term opportunities in this
>>> case. Consider you register a Spark temp view as some sort of data frame
>>> read, then it could still be resolved to a Spark plan that is representable
>>> by an intermediate representation. But I agree this gets very complicated
>>> very soon, and just having the case (1) covered would already be a huge
>>> step forward.
>>>
>>> -Jack
>>>
>>>
>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <btc...@gmail.com> wrote:
>>>
>>>> It's interesting to note that a tabular SQL UDF can be used to build a 
>>>> *parameterized
>>>> *view.  So, there's definitely a lot in common between UDFs and views.
>>>>
>>>> Thanks
>>>>
>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin Moustafa <
>>>> wa.moust...@gmail.com> wrote:
>>>>
>>>>> I think there is a disconnect about what is perceived as a "UDF".
>>>>> There are 2 flavors:
>>>>>
>>>>> (1) Functions that are defined by the user whose definition is a
>>>>> composition of other built-in functions/SQL expressions.
>>>>> (2) Custom code written in imperative function according to a
>>>>> Java/Scala/Python API, etc.
>>>>>
>>>>> All the examples in Ajantha's references are pretty much from (1) and
>>>>> I think those have more analogy to views due to their SQL nature. Agree 
>>>>> (2)
>>>>> is not practical to maintain by Iceberg, but I think Ajantha's use cases
>>>>> are around (1), and may be worth evaluating.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>>
>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <ajanthab...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I guess we'll know more when you post the proposal, but I think this
>>>>>>> would be a very difficult area to tackle across engines, languages, and
>>>>>>> memory models without having a huge performance penalty.
>>>>>>
>>>>>> Assuming Iceberg initially supports SQL representations of UDFs
>>>>>> (similar to views as shared by the reference links above), the complexity
>>>>>> involved will be similar to managing views.
>>>>>>
>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>> We will work on publishing the draft spec (inspired by the view spec)
>>>>>> this week to facilitate further discussions.
>>>>>>
>>>>>> - Ajantha
>>>>>>
>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <yezhao...@gmail.com> wrote:
>>>>>>
>>>>>>> > While it would be great to have a common set of functions across
>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>> user-supplied code -- seems inherently specialized to me and should be 
>>>>>>> part
>>>>>>> of the engines' design.
>>>>>>>
>>>>>>> How is this different from the views? I feel we can say exactly the
>>>>>>> same thing for Iceberg views, but yet we have Iceberg multi-dialect 
>>>>>>> views
>>>>>>> implemented. Maybe it sounds like we are trying to draw a line between 
>>>>>>> SQL
>>>>>>> vs other programming language as "code"? but I think SQL is just another
>>>>>>> type of code, and we are already talking about compiling all these
>>>>>>> different code dialects to an intermediate representation (using 
>>>>>>> projects
>>>>>>> like Coral, Substrait), which will be stored as another type of
>>>>>>> representation of Iceberg view. I think the same functionality can be 
>>>>>>> used
>>>>>>> for UDFs if developed.
>>>>>>>
>>>>>>> I actually hink adding UDF support is a good idea, even just a
>>>>>>> multi-dialect one like view, and that can allow engines to for example
>>>>>>> parse a view SQL, and when a function referenced cannot be resolved, 
>>>>>>> try to
>>>>>>> seek for a multi-dialect UDF definition.
>>>>>>>
>>>>>>> I guess we can discuss more when we have the actual proposal
>>>>>>> published.
>>>>>>>
>>>>>>> Best,
>>>>>>> Jack Ye
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>
>>>>>>>> UDFs are as engine specific and portable and "non-centralized" as
>>>>>>>> views are. The same performance concerns apply to views as well.
>>>>>>>> Iceberg should define a common base upon which engines can build,
>>>>>>>> so the argument that UDFs aren't practical, because engines are 
>>>>>>>> different,
>>>>>>>> is probably only a temporary concern.
>>>>>>>>
>>>>>>>> In the long term, Iceberg should also try to tackle the idea to
>>>>>>>> make views portable, which is conceptually not that much different from
>>>>>>>> portable UDFs.
>>>>>>>>
>>>>>>>>
>>>>>>>> PS: I'm not a fan of adding a negative touch to the idea of having
>>>>>>>> UDFs in Iceberg, especially not in this early stage.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>
>>>>>>>> Thanks, Ajantha.
>>>>>>>>
>>>>>>>> I'm skeptical about whether it's a good idea to add UDFs tracked by
>>>>>>>> Iceberg catalogs. I think that Iceberg primarily deals with things 
>>>>>>>> that are
>>>>>>>> centralized, like tables of data. While it would be great to have a 
>>>>>>>> common
>>>>>>>> set of functions across engines, I don't see how that is practical when
>>>>>>>> those engines are implemented so differently. Plugging in code -- and
>>>>>>>> especially custom user-supplied code -- seems inherently specialized 
>>>>>>>> to me
>>>>>>>> and should be part of the engines' design.
>>>>>>>>
>>>>>>>> I guess we'll know more when you post the proposal, but I think
>>>>>>>> this would be a very difficult area to tackle across engines, 
>>>>>>>> languages,
>>>>>>>> and memory models without having a huge performance penalty.
>>>>>>>>
>>>>>>>> Ryan
>>>>>>>>
>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <ajanthab...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Everyone,
>>>>>>>>>
>>>>>>>>> This is a discussion to gauge the community interest in storing
>>>>>>>>> the Versioned SQL UDFs in Iceberg.
>>>>>>>>> We want to propose the spec addition for storing the versioned
>>>>>>>>> UDFs in Iceberg (inspired by view spec).
>>>>>>>>>
>>>>>>>>> These UDFs can operate similarly to views in that they are
>>>>>>>>> associated with tables, but they can accept arguments and produce 
>>>>>>>>> return
>>>>>>>>> values, or even function as inline expressions.
>>>>>>>>> Many Query engines like Dremio, Trino, Snowflake, Databricks Spark
>>>>>>>>> supports SQL UDFs at catalog level [1].
>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>> - Interoperability between the engines. Potentially engines can
>>>>>>>>> understand the UDFs written by other engines (with the translate 
>>>>>>>>> layer).
>>>>>>>>>
>>>>>>>>> We believe that integrating this feature into Iceberg would be a
>>>>>>>>> valuable addition, and we're eager to collaborate with the community 
>>>>>>>>> to
>>>>>>>>> develop a UDF specification.
>>>>>>>>> Stephen <stephen....@dremio.com> has already begun drafting a
>>>>>>>>> specification to propose to the community.
>>>>>>>>>
>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> Dremio -
>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>> Trino - https://trino.io/docs/current/sql/create-function.html
>>>>>>>>> Snowflake -
>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>> Databricks -
>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>> --
>>>>>>>> Robert Stupp
>>>>>>>> @snazy
>>>>>>>>
>>>>>>>>

Reply via email to