Hi,
as far as I know these are not standard functions.
Writing UDFs is easy, but only in Java and Scala is it equally efficient as a
built-in function. When using Python, data movement/conversion to/from Arrow is
still necessary, and that makes a difference in performance. That was the
motiva
Is it standard SQL or implemented in Hive? Because UDFs are so relatively
easy in Spark we don't need tons of builtins like an RDBMS does.
On Tue, Feb 5, 2019, 7:43 AM Petar Zečević
> Hi everybody,
> I finally created the JIRA ticket and the pull request for the two array
> indexing functions:
>
Hi everybody,
I finally created the JIRA ticket and the pull request for the two array
indexing functions:
https://issues.apache.org/jira/browse/SPARK-26826
Can any of the committers please check it out?
Thanks,
Petar
Petar Zečević writes:
> Hi,
> I implemented two array functions that are
Hi,
yes, these are imlemented just like native functions in sql.functions, with
code generation, so whole-stage codegen should apply.
Regarding plan optimization, I am not sure how these would be taken into
account in the existing rules, except maybe for filter pushdown.
Petar
Alessandro So
Hi Petar,
I have implemented similar functions a few times through ad-hoc UDFs in the
past, so +1 from me.
Can you elaborate a bit more on how you practically implement those
functions? Are they UDF or "native" functions like those in sql.functions
package?
I am asking because I wonder if/how Cat
