Indeed. But nested aggregation does not work with Structured Streaming, that's the point. I would like to know if there is workaround, or what's the plan regarding this feature which seems to me quite useful. If the implementation is not overtly complex and it is just a matter of manpower, I am fine with devoting some time to it.
-- Arnaud Bailly twitter: abailly skype: arnaud-bailly linkedin: http://fr.linkedin.com/in/arnaudbailly/ On Thu, Jul 7, 2016 at 2:17 PM, Sivakumaran S <siva.kuma...@me.com> wrote: > Arnauld, > > You could aggregate the first table and then merge it with the second > table (assuming that they are similarly structured) and then carry out the > second aggregation. Unless the data is very large, I don’t see why you > should persist it to disk. IMO, nested aggregation is more elegant and > readable than a complex single stage. > > Regards, > > Sivakumaran > > > > On 07-Jul-2016, at 1:06 PM, Arnaud Bailly <arnaud.oq...@gmail.com> wrote: > > It's aggregation at multiple levels in a query: first do some aggregation > on one tavle, then join with another table and do a second aggregation. I > could probably rewrite the query in such a way that it does aggregation in > one pass but that would obfuscate the purpose of the various stages. > Le 7 juil. 2016 12:55, "Sivakumaran S" <siva.kuma...@me.com> a écrit : > >> Hi Arnauld, >> >> Sorry for the doubt, but what exactly is multiple aggregation? What is >> the use case? >> >> Regards, >> >> Sivakumaran >> >> >> On 07-Jul-2016, at 11:18 AM, Arnaud Bailly <arnaud.oq...@gmail.com> >> wrote: >> >> Hello, >> >> I understand multiple aggregations over streaming dataframes is not >> currently supported in Spark 2.0. Is there a workaround? Out of the top of >> my head I could think of having a two stage approach: >> - first query writes output to disk/memory using "complete" mode >> - second query reads from this output >> >> Does this makes sense? >> >> Furthermore, I would like to understand what are the technical hurdles >> that are preventing Spark SQL from implementing multiple aggregation right >> now? >> >> Thanks, >> -- >> Arnaud Bailly >> >> twitter: abailly >> skype: arnaud-bailly >> linkedin: http://fr.linkedin.com/in/arnaudbailly/ >> >> >> >
