Hi Philipp, When used against Kinesalite, can you tell if the connector is already reading data from the test shard before any of the shard discovery messages? If you have any spare time to test this, you can set a larger value for the `ConsumerConfigConstants.SHARD_DISCOVERY_INTERVAL_MILLIS` in the config properties to easier test this.
If yes, I’m suspecting the problem is that Kinesalite hasn’t sticked to the actual Kinesis behaviour for some of the APIs. Specifically, I think the problem is with the `describeStream(streamName, lastSeenShardId)` Kinesis API, where the expected behaviour is that the returned shard list only contains shardIds after `lastSeenShardId`. Perhaps Kinesalite didn’t follow the behaviour on this part. That’s why the connector kept on determining that it’s a new discovered shard. I’ll investigate and try to reproduce the problem, and see if there’s a good way to workaround this for Kinesalite. Thank you for reporting the issue, I’ve filed up a JIRA (https://issues.apache.org/jira/browse/FLINK-5075) for this. Best, Gordon On November 16, 2016 at 5:03:17 AM, Philipp Bussche (philipp.buss...@gmail.com) wrote: has discovered a new shard KinesisStreamShard
