[structured streaming] How to remove outdated data when use Window Operations

2016-11-29 Thread Xinyu Zhang
Hi I want to use window operations. However, if i don't remove any data, the "complete" table will become larger and larger as time goes on. So I want to remove some outdated data in the complete table that I would never use. Is there any method to meet my requirement? Thanks!

Re:Re: Re: Multiple streaming aggregations in structured streaming

2016-11-22 Thread Xinyu Zhang
Could you please tell me how to use the approximate count distinct? Is there any docs? Thanks At 2016-11-21 15:56:21, "Reynold Xin" wrote: Can you use the approximate count distinct? On Sun, Nov 20, 2016 at 11:51 PM, Xinyu Zhang wrote: MapWithState is also very useful.

Re:Re: Multiple streaming aggregations in structured streaming

2016-11-20 Thread Xinyu Zhang
MapWithState is also very useful. I want to calculate UV in real time, but "distinct count" and "multiple streaming aggregations" are not supported. Is there any method to calculate real-time UV in the current version? At 2016-11-19 06:01:45, "Michael Armbrust" wrote: Doing this generally