Re: Systematic error when re-starting Spark stream unless I delete all checkpoints

2014-09-30 Thread Svend Vanderveken
Hi again, Just FYI, I found the mistake in my code regarding restartability of spark streaming: I had a method providing a context (either retrieved from checkpoint or, if no checkpoint available, built anew) and was building then starting a stream on it. The mistake is that we should not build

Re: Systematic error when re-starting Spark stream unless I delete all checkpoints

2014-09-26 Thread Svend Vanderveken
Hi all, I apologise for re-posting this, I realise some mail systems are filtering all the code samples from the original post. I would greatly appreciate any pointer regarding, this issue basically renders spark streaming not fault-tolerant for us. Thanks in advance, S --- " I experience s