Re: spark streaming window operations on a large window size

2015-02-24 Thread Avi Levi
OK - thanks a lot On Tue, Feb 24, 2015 at 9:49 AM, Tathagata Das wrote: > Yes. > > On Mon, Feb 23, 2015 at 11:16 PM, Avi Levi wrote: > >> @Tathagata Das so basically you are saying it is supported out of the >> box, but we should expect a significant perform

Re: spark streaming window operations on a large window size

2015-02-23 Thread Avi Levi
@Tathagata Das so basically you are saying it is supported out of the box, but we should expect a significant performance hit - is that right? On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das wrote: > The default persistence level is MEMORY_AND_DISK, so the LRU policy would > discard the blocks