Issue while trying to aggregate with a sliding window

Trying to aggregate over a sliding window, playing with the slide duration. Playing around with the slide interval I can see the aggregation works but mostly fails with the below error. The stream has records coming in at 100ms. JavaPairDStream aggregatedDStream = pairsDStream.reduceByKeyAndWindow

Re: Issue while trying to aggregate with a sliding window

Thanks! Will try to get the fix and retest. On Tue, Jun 17, 2014 at 5:30 PM, onpoq l wrote: > There is a bug: > > https://github.com/apache/spark/pull/961#issuecomment-45125185 > > > On Tue, Jun 17, 2014 at 8:19 PM, Hatch M wrote: > > Trying to aggregate over a slid

Re: Issue while trying to aggregate with a sliding window

duration ? Shouldn't the reduceByKeyAndWindow aggregate every record in a given window (zeroTime to zeroTime+windowDuration)? On Tue, Jun 17, 2014 at 10:55 PM, Hatch M wrote: > Thanks! Will try to get the fix and retest. > > > On Tue, Jun 17, 2014 at 5:30 PM, onpoq l wrote: &g

Spark Zmq issue in cluster mode

I have a spark streaming zmq application running fine in non-cluster mode. When running a local cluster and I do spark-submit, zero mq java client is choking. org.zeromq.ZMQException: No such file or directory at org.zeromq.ZMQ$Socket.raiseZMQException(ZMQ.java:480) at org.zeromq.ZMQ$Socket.recv(Z