Re: Window slide duration

2014-06-03 Thread Vadim Chekan
Лучше по частям собрать. http://www.newegg.com/Product/Product.aspx?Item=N82E16813157497 Пассивное охлаждение, 16Гб памяти можно поставить. А на то что ты прислал 4Гб максимум, это не годиться. Выбрать малый корпус и дело с концом. On Tue, Jun 3, 2014 at 4:35 PM, Vadim Chekan wrote: > Ok, i

Re: Window slide duration

2014-06-03 Thread Vadim Chekan
Ok, it's a bug in spark. I've submitted a patch: https://issues.apache.org/jira/browse/SPARK-2009 On Mon, Jun 2, 2014 at 8:39 PM, Vadim Chekan wrote: > Thanks for looking into this Tathagata. > > Are you looking for traces of ReceiveInputDStream.clearMetadata call? >

Re: Window slide duration

2014-06-02 Thread Vadim Chekan
key " > 1401754908000 > ms" > > TD > > > On Mon, Jun 2, 2014 at 5:38 PM, Vadim Chekan > wrote: > >> Ok, it seems like "Time ... is invalid" is part of normal workflow, when >> window DStream will ignore RDDs at moments in time when they do not

Re: Window slide duration

2014-06-02 Thread Vadim Chekan
ue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) On Mon, Jun 2, 2014 at 5:22 PM, Vadim Chekan wrote: > Hi all, > > I

Window slide duration

2014-06-02 Thread Vadim Chekan
Hi all, I am getting an error: 14/06/02 17:06:32 INFO WindowedDStream: Time 1401753992000 ms is invalid as zeroTime is 1401753986000 ms and slideDuration is 4000 ms and difference is 6000 ms 14/06/02 17:06:32 ERROR OneForOneStrategy: key not found: 1401753992000 ms ===

Checkpoint serialization

2014-05-19 Thread Vadim Chekan
Hi all, I'm writing an spark streaming app which consumes Kafka stream. Objects consumed are Avro objects (they do not implement java serialization). I decided to use Kryo and have set "spark.serializer" and "spark.closure.serializer". But now I am getting exception in CheckpointWriter: 14/05/19 1