Re: Does Rollups work with spark structured streaming with state.

Great Amit, best of luck Cheers, Mich view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on th

Re: Does Rollups work with spark structured streaming with state.

HI Mich, Thanks for your email. I have tried for the batch mode, Still looking to try in streaming mode. Will update you as per. Regards Amit Joshi On Thu, Jun 17, 2021 at 1:07 PM Mich Talebzadeh wrote: > OK let us start with the basic cube > > create a DF first > > scala> val df = Seq( >

Re: Does Rollups work with spark structured streaming with state.

OK let us start with the basic cube create a DF first scala> val df = Seq( | ("bar", 2L), | ("bar", 2L), | ("foo", 1L), | ("foo", 2L) | ).toDF("word", "num") df: org.apache.spark.sql.DataFrame = [word: string, num: bigint] Now try cube on it scala> df.cube($"w

Re: Does Rollups work with spark structured streaming with state.

Hi Mich, Yes, you may think of cube rollups. Let me try to give an example: If we have a stream of data like (country,area,count, time), we would be able to get the updated count with different combinations of keys. > As example - > (country - count) > (country , area - count) We may need to

Re: Does Rollups work with spark structured streaming with state.

Hi, Just to clarify Are we talking about* rollup* as a subset of a cube that computes hierarchical subtotals from left to right? view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any