Re: Converting RelationalGroupedDataSet to DataFrame

2021-02-07 Thread Stephane Verlet
Once you have a RelationalGroupedDataSet , you can use agg() to perform group wide operation such max , sum , etc ... or even custom aggregator. df.groupBy().agg(sum(col(...))) That will return a DF with your groupBy columns and result of the aggregation Stephane Soheil Pourbafrani wrote: Hi,

Converting RelationalGroupedDataSet to DataFrame

2021-02-06 Thread Soheil Pourbafrani
Hi, In my problem, I need to group the DataFrame, apply the business logic for each group and finally emit a new DataFrame based on that. To describe in detail, there is a device_dataframe which contains the timestamp of when the device had been turned on (on) and turned off (off). +-+---