Does spark streaming consumer for kinesis uses Kinesis Client Library and mandates to checkpoint the sequence number of shards in dynamo db.
Will it lead to dataloss if consumed datarecords are not yet processed and kinesis checkpointed the consumed sequenece numbers in dynamo db and spark worker crashes - then spark launched the worker on another node but start consuming from dynamo db's checkpointed sequence number which is ahead of processed sequenece number . is there a way to checkpoint the sequenece numbers ourselves in Kinesis as it is in Kafka low level consumer ? Thanks
