Re: Persisting DStreams

2015-11-16 Thread Jean-Baptiste Onofré
Hi Fernando, the "persistence" of a DStream is defined depending of the StorageLevel. Window is not related to persistence: it's the processing of multiple DStream in one, a kind of "gather of DStreams". The transformation is applied on a "slide window". For instance, you define a window of 3

Persisting DStreams

2015-11-16 Thread Fernando O.
Hi all, I was wondering if someone could give me a brief explanation or point me in the right direction in the code for where DStream persistence is donde. I'm looking at DStream.java but all it does is setting the StorageLevel, and neither WindowedDStream or ReducedWindowedDStream seem to chang