I'm not exactly sure about the receiver you pointed though, if you point the "KinesisReceiver" implementation, yes.
Also, we currently cannot disable the interval checkpoints. On Tue, Oct 25, 2016 at 11:53 AM, Shushant Arora <shushantaror...@gmail.com> wrote: > Thanks! > > Is kinesis streams are receiver based only? Is there non receiver based > consumer for Kinesis ? > > And Instead of having fixed checkpoint interval,Can I disable auto > checkpoint and say when my worker has processed the data after last record > of mapPartition now checkpoint the sequence no using some api. > > > > On Tue, Oct 25, 2016 at 7:07 AM, Takeshi Yamamuro <linguin....@gmail.com> > wrote: > >> Hi, >> >> The only thing you can do for Kinesis checkpoints is tune the interval of >> them. >> https://github.com/apache/spark/blob/master/external/kinesis >> -asl/src/main/scala/org/apache/spark/streaming/kinesis/ >> KinesisUtils.scala#L68 >> >> Whether the dataloss occurs or not depends on the storage level you set; >> if you set StorageLevel.MEMORY_AND_DISK_2, Spark may continue processing >> in case of the dataloss because the stream data Spark receives are >> replicated across executors. >> However, all the executors that have the replicated data crash, >> IIUC the dataloss occurs. >> >> // maropu >> >> On Mon, Oct 24, 2016 at 4:43 PM, Shushant Arora < >> shushantaror...@gmail.com> wrote: >> >>> Does spark streaming consumer for kinesis uses Kinesis Client Library >>> and mandates to checkpoint the sequence number of shards in dynamo db. >>> >>> Will it lead to dataloss if consumed datarecords are not yet processed >>> and kinesis checkpointed the consumed sequenece numbers in dynamo db and >>> spark worker crashes - then spark launched the worker on another node but >>> start consuming from dynamo db's checkpointed sequence number which is >>> ahead of processed sequenece number . >>> >>> is there a way to checkpoint the sequenece numbers ourselves in Kinesis >>> as it is in Kafka low level consumer ? >>> >>> Thanks >>> >>> >> >> >> -- >> --- >> Takeshi Yamamuro >> > > -- --- Takeshi Yamamuro
