Re: Consuming data from dynamoDB streams to flink

2019-08-08 Thread Vinay Patil
Hello, For anyone looking for setting up alerts for flink application ,here is good blog by Flink itself : https://www.ververica.com/blog/monitoring-apache-flink-applications-101 So, for dynamoDb streams we can set the alert on millisBehindLatest Regards, Vinay Patil On Wed, Aug 7, 2019 at 2:24

Re: Consuming data from dynamoDB streams to flink

2019-08-07 Thread Vinay Patil
Hi Andrey, Thank you for your reply, I understand that the checkpoints are gone when the job is cancelled or killed, may be configuring external checkpoints will help here so that we can resume from there. My points was if the job is terminated, and the stream position is set to TRIM_HORIZON , th

Re: Consuming data from dynamoDB streams to flink

2019-07-19 Thread Andrey Zagrebin
Hi Vinay, 1. I would assume it works similar to kinesis connector (correct me if wrong, people who actually developed it) 2. If you have activated just checkpointing, the checkpoints are gone if you externally kill the job. You might be interested in savepoints [1] 3. See paragraph in [2] about ki

Re: Consuming data from dynamoDB streams to flink

2019-07-19 Thread Vinay Patil
Hi, I am using this consumer for processing records from DynamoDb Streams , few questions on this : 1. How does checkpointing works with Dstreams, since this class is extending FlinkKinesisConsumer, I am assuming it will start from the last successful checkpoint in case of failure, right ? 2. Cur