Tzu-Li (Gordon) Tai created FLINK-3231:
------------------------------------------
Summary: Handle Kinesis-side resharding in Kinesis streaming
consumer
Key: FLINK-3231
URL: https://issues.apache.org/jira/browse/FLINK-3231
Project: Flink
Issue Type: Sub-task
Components: Streaming Connectors
Affects Versions: 1.0.0
Reporter: Tzu-Li (Gordon) Tai
A big difference between Kinesis shards and Kafka partitions is that Kinesis
users can choose to "merge" and "split" shards at any time for adjustable
stream throughput capacity. This article explains this quite clearly:
https://brandur.org/kinesis-by-example.
This will break the static shard-to-task mapping implemented in the basic
version of the Kinesis consumer
(https://issues.apache.org/jira/browse/FLINK-3211). The static shard-to-task
mapping is done in a simple round-robin-like distribution which can be locally
determined at each Flink consumer task (Flink Kafka consumer does this too).
To handle Kinesis resharding, we will need some way to let the Flink consumer
tasks coordinate which shards they are currently handling, and allow the tasks
to ask the coordinator for a shards reassignment when the task finds out it has
found a closed shard at runtime (shards will be closed by Kinesis when it is
merged and split).
A possible approach to this is a centralized coordinator state store which is
visible to all Flink consumer tasks. Tasks can use this state store to locally
determine what shards it can be reassigned. Zookeeper can be used for this
state store, but that means it would require the user to set up ZK to work.
Since this feature introduces extensive work, it is opened as a separate
sub-task from the basic implementation
https://issues.apache.org/jira/browse/FLINK-3211.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
