[
https://issues.apache.org/jira/browse/FLINK-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346257#comment-16346257
]
ASF GitHub Bot commented on FLINK-8516:
---------------------------------------
Github user tweise commented on a diff in the pull request:
https://github.com/apache/flink/pull/5393#discussion_r164952695
--- Diff:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java
---
@@ -93,6 +93,12 @@
/** User supplied deserialization schema to convert Kinesis byte
messages to Flink objects. */
private final KinesisDeserializationSchema<T> deserializer;
+ /**
+ * The function that determines which subtask a shard should be
assigned to.
+ */
+ // TODO: instead of the property, use a factory method that would allow
subclass to access source context?
--- End diff --
createFn(...) that will allow the function to be created with access to
runtime context (like the number of subtasks), and then change the fn signature
to only take shard metadata as parameter. Subclasses can override createFn,
instead of having the property.
> FlinkKinesisConsumer does not balance shards over subtasks
> ----------------------------------------------------------
>
> Key: FLINK-8516
> URL: https://issues.apache.org/jira/browse/FLINK-8516
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Affects Versions: 1.4.0, 1.3.2, 1.5.0
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Priority: Major
>
> The hash code of the shard is used to distribute discovered shards over
> subtasks round robin. This works as long as shard identifiers are sequential.
> After shards are rebalanced in Kinesis, that may no longer be the case and
> the distribution become skewed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
