RE: [SQL] Is it worth it (and advisable) to implement native UDFs?

2020-02-04 Thread email
Is there any documentation/ sample about this besides the pull requests merged to spark core? It seems that I need to create my custom functions under the package org.apache.spark.sql.* in order to be able to access some of the internal classes I saw in[1] such as Column[2] Could you ple

[SQL] Is it worth it (and advisable) to implement native UDFs?

2020-01-20 Thread email
Hi, I read online[1] that for a best UDF performance it is possible to implement them using internal Spark expressions, and I also saw a couple of pull requests such as [2] and [3] where this was put to practice (not sure if for that reason or just to extend the API). We have an algorithm