On Sun, Mar 4, 2018 at 11:38 PM, Bowen Li <bowenl...@gmail.com> wrote:
> If ListShards() gives all the info that Flink needs, +1 on switching. > DescribeStreams() has a limitation of 5 requests/sec, which is pretty > bad.... > > But, I believe the goal of switching APIs should be *making Flink jobs that > read from Kinesis more stable*, rather than having faster shard discovery > rate. The default shard discovery rate is every 10s, which is already very > very fast and can satisfy most Kinesis users, we shouldn't shorten the > default value anymore. Developers who want faster discovery rate than 10s > should overwrite the default value themselves. > > https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html "This operation has a limit of 10 transactions per second per account." 10s interval would be sufficient, but with 5 requests per second and a larger number of subtasks and multiple applications in the account, we are looking at discovery rates in double digit minutes. That's where ListShards will help. Of course, the consumer should still be redesigned to centralize the discovery. Thanks, Thomas
