Re: FlinkKinesis consumer

2021-07-26 Thread Danny Cranmer
Hello, If you are consuming from a single stream, you can use the shard ID to achieve a better distribution. Since the shard IDs are assigned incrementally like so: - shardId- - shardId-0001 - shardId-0002 - etc You can substring the prefix and convert

Re: FlinkKinesis consumer

2021-07-25 Thread Caizhi Weng
Hi! It's stated on the line just below that in the document. It is recommended to monitor the shard distribution and adjust assignment > appropriately. A custom assigner implementation can be set via > setShardAssigner(KinesisShardAssigner) to optimize the hash function or use > static overrides

FlinkKinesis consumer

2021-07-23 Thread Vijayendra Yadav
Hi Team, https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.html *" There is no perfect generic default assignment function. Default shard to subtask assignment, which is based on hash code, may result in skew,