Hi, https://stackoverflow.com/questions/32100973/how-to-define-and-use-a-user-defined-aggregate-function-in-spark-sql has a good overview and the best sample I have found so far. (besides spark source code).
Best, Georg Am Mi., 23. Jan. 2019 um 17:16 Uhr schrieb Georg Heiler < georg.kf.hei...@gmail.com>: > Hi Herman, > > Thanks a lot. So far I only found most of the documentation about UDAF. > Could you point me anywhere (besides just reading spark's source code) > which explains how to work with custom AggregateFunctions? > > Best, > Georg > > Am Mi., 23. Jan. 2019 um 16:02 Uhr schrieb Herman van Hovell < > her...@databricks.com>: > >> Hi Georg, >> >> In most cases you want to implement an aggregate function. You can either >> define a UDAF, or an AggregateFunction. The latter is an internal >> expression, so it is faster but we might break it across versions of spark. >> >> Hope this helps. >> >> Cheers, >> Herman >> >> On Wed, Jan 23, 2019 at 3:55 PM Georg Heiler <georg.kf.hei...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I want to write custom window functions in spark which are also >>> optimisable for catalyst. >>> Can you provide some hints where to start? >>> >>> Also posting to DEVLIST as I believe this is a rather exotic topic. >>> >>> Best, >>> Georg >>> >>
