Re: Pyspark define UDF for windows

2017-09-20 Thread Weichen Xu
UDF cannot be used as window function. You can use built-in window function or UDAF. On Wed, Sep 20, 2017 at 7:23 PM, Simon Dirmeier wrote: > Dear all, > I am trying to partition a DataFrame into windows and then for every > column and window use a custom function (udf) using Spark's Python > in

Pyspark define UDF for windows

2017-09-20 Thread Simon Dirmeier
Dear all, I am trying to partition a DataFrame into windows and then for every column and window use a custom function (udf) using Spark's Python interface. Within that function I cast a column of a window in a m x n matrix to do a median-polish and afterwards return a list again. This doesn