OK - thanks a lot
On Tue, Feb 24, 2015 at 9:49 AM, Tathagata Das wrote:
> Yes.
>
> On Mon, Feb 23, 2015 at 11:16 PM, Avi Levi wrote:
>
>> @Tathagata Das so basically you are saying it is supported out of the
>> box, but we should expect a significant performance hit - is that right?
>>
>>
>>
>>
Yes.
On Mon, Feb 23, 2015 at 11:16 PM, Avi Levi wrote:
> @Tathagata Das so basically you are saying it is supported out of the
> box, but we should expect a significant performance hit - is that right?
>
>
>
> On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das
> wrote:
>
>> The default persistence
@Tathagata Das so basically you are saying it is supported out of the box,
but we should expect a significant performance hit - is that right?
On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das wrote:
> The default persistence level is MEMORY_AND_DISK, so the LRU policy would
> discard the blocks
The default persistence level is MEMORY_AND_DISK, so the LRU policy would
discard the blocks to disk, so the streaming app will not fail. However,
since things will get constantly read in and out of disk as windows are
processed, the performance wont be great. So it is best to have sufficient
memor
I don't think current Spark Streaming supports window operations which beyond
its available memory, internally Spark Streaming puts all the data in the
memory belongs to the effective window, if the memory is not enough,
BlockManager will discard the blocks at LRU policy, so something unexpected
