Hi there,
I am looking into AWS Kinesis and wanted to test with a local install of
Kinesalite. This is on the Flink 1.2-SNAPSHOT. However it looks like my
subtask keeps on discovering new shards indicated by the following log
messages which is constantly written:
21:45:42,867 INFO
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher
- Subtask 0 has discovered a new shard
KinesisStreamShard{streamName='TestKinesisStream', shard='{ShardId:
shardId-000000000000,HashKeyRange: {StartingHashKey: 0,EndingHashKey:
340282366920938463463374607431768211455},SequenceNumberRange:
{StartingSequenceNumber:
49567668736415295912662799644604754736261244425831710722,}}'} due to
resharding, and will start consuming the shard from sequence number
EARLIEST_SEQUENCE_NUM with ShardConsumer 4
There is a new one discovered every 10 seconds or so. I have set parallism
to 1 and checkpointing to 5000ms.
This does not happen when I am running directly against AWS.
I wonder if this is really something inside Flink or rather Kinesalite,
however the part of the code in the Flink connector where the respective log
message is written is heavily documented with FLINK-4341 so I wondering if
this is maybe a side effect ?
Thanks
Philipp
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Subtask-keeps-on-discovering-new-Kinesis-shard-when-using-Kinesalite-tp10133.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
