Re: Issue while trying to aggregate with a sliding window

2014-06-17 Thread onpoq l
There is a bug: https://github.com/apache/spark/pull/961#issuecomment-45125185 On Tue, Jun 17, 2014 at 8:19 PM, Hatch M wrote: > Trying to aggregate over a sliding window, playing with the slide duration. > Playing around with the slide interval I can see the aggregation works but > mostly fail

shuffling using netty in spark streaming

2014-06-11 Thread onpoq l
Hi, 1. Does netty perform better than the basic method for shuffling? I found the latency caused by shuffling in a streaming job is not stable with the basic method. 2. However, after I turn on netty for shuffling, I can only see the results for the first two batches, and then no output at all. I