[
https://issues.apache.org/jira/browse/FLINK-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867437#comment-15867437
]
Fabian Hueske commented on FLINK-5802:
--------------------------------------
Thanks for opening this issue [~clarkyzl].
I agree, supporting Hive UDFs would be a great feature for the Table API.
Since there are several ways to achieve this, we should discuss the design
first.
>From the top of my head I can think of two approaches:
1. Native support by extending the internals of the Table API. This would mean
that we have Hive specific code to register functions and integrate them with
code generation. Depending on the interface of the Hive UDFs it might even mean
that we have to generate a different physical execution plan. It also means
that we will have a dependency on Hive in the Table API which I personally
would like to avoid.
2. Support by wrapping. For this, we would implement Table API UDFs that
internally wrap Hive UDFs. Since the wrappers are treated like regular Table
API UDFs, we do not need to change the internals of the Table API. On the other
hand this does only work if the interfaces of Table API UDFs and Hive UDFs are
compatible (if they are not we would probably need different execution plans).
Another pro aspect is that the wrappers could be located in a separate Maven
module which also prevents a hard Hive dependency in flink-table.
I would opt for the second approach because it has less implications on the
internals of the Table API. If we figure out that this approach will not work
for some Hive UDFs we have to decide whether it is worth to support those or
not.
> Flink SQL calling Hive User-Defined Functions
> ---------------------------------------------
>
> Key: FLINK-5802
> URL: https://issues.apache.org/jira/browse/FLINK-5802
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
> Reporter: Zhuoluo Yang
> Labels: features
>
> It's important to call hive udf in Flink SQL. A great many udfs were written
> in hive since last ten years.
> It's really important to reuse the hive udfs. This feature will reduce the
> cost of migration and bring more users to flink.
> Spark SQL has already supported this function.
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_spark-guide/content/calling-udfs.html
> The Hive UDFs here include both built-in UDFs and customized UDFs. As many
> business logic had been written in UDFs, the customized UDFs are more
> important than the built-in UDFs.
> Generally, there are three kinds of UDFs in Hive: UDF, UDTF and UDAF.
> Here is the document of the Spark SQL:
> http://spark.apache.org/docs/latest/sql-programming-guide.html#compatibility-with-apache-hive
>
> Spark code:
> https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
> https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
