Ohh that explains the reason.
My use case does not need state management.
So i guess i am better off without checkpointing.
Thank you for clarification.
Regards,
Chandan
On Sat, Aug 20, 2016 at 8:24 AM, Cody Koeninger wrote:
> Checkpointing is required to be turned on in certain situations (e.g
Checkpointing is required to be turned on in certain situations (e.g.
updateStateByKey), but you're certainly not required to rely on it for
fault tolerance. I try not to.
On Fri, Aug 19, 2016 at 1:51 PM, chandan prakash
wrote:
> Thanks Cody for the pointer.
>
> I am able to do this now. Not us
Thanks Cody for the pointer.
I am able to do this now. Not using checkpointing. Rather storing offsets
in zookeeper for fault tolerance.
Spark Config changes now getting reflected in code deployment.
*Using this api :*
*KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder, (S
Yes,
i looked into the source code implementation. sparkConf is serialized and
saved during checkpointing and re-created from the checkpoint directory at
time of restart. So any sparkConf parameter which you load from
application.config and set in sparkConf object in code cannot be changed
and re
Checkpointing is not kafka-specific. It encompasses metadata about the
application. You can't re-use a checkpoint if your application has changed.
http://spark.apache.org/docs/latest/streaming-programming-guide.html#upgrading-application-code
On Thu, Aug 18, 2016 at 4:39 AM, chandan prakash
w
Is it possible that i use checkpoint directory to restart streaming but
with modified parameter value in config file (e.g. username/password for
db connection) ?
Thanks in advance.
Regards,
Chandan
On Thu, Aug 18, 2016 at 1:10 PM, chandan prakash
wrote:
> Hi,
> I am using direct kafka with ch
