[
https://issues.apache.org/jira/browse/KAFKA-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Clemens Valiente updated KAFKA-6391:
------------------------------------
Description:
https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L394
Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions is created
from repartitionTopicMetadata
THEN we do ensureCoPartitioning on repartitionTopicMetadata
THEN we create topics and partitions according to repartitionTopicMetadata
THEN we use allRepartitionTopicPartitions to create our Cluster fullMetadata
THEN we use fullMetadata to assign the tasks and no longer use
repartitionTopicMetadata
This results in any change to repartitionTopicMetadata in ensureCoPartitioning
to be used for creating partitions but no tasks are ever created for any
partition added by ensureCoPartitioning()
the fix is easy: First ensureCoPartitioning() on repartitionTopicMetadata
before creating allRepartitionTopicPartitions
was:
https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L366
Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions is created
from repartitionTopicMetadata
THEN we do ensureCoPartitioning on repartitionTopicMetadata
THEN we create topics and partitions according to repartitionTopicMetadata
THEN we use allRepartitionTopicPartitions to create our Cluster fullMetadata
THEN we use fullMetadata to assign the tasks and no longer use
repartitionTopicMetadata
This results in any change to repartitionTopicMetadata in ensureCoPartitioning
to be used for creating partitions but no tasks are ever created for any
partition added by ensureCoPartitioning()
the fix is easy: First ensureCoPartitioning() on repartitionTopicMetadata
before creating allRepartitionTopicPartitions
> output from ensure copartitioning is not used for Cluster metadata
> ------------------------------------------------------------------
>
> Key: KAFKA-6391
> URL: https://issues.apache.org/jira/browse/KAFKA-6391
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 1.0.0
> Reporter: Clemens Valiente
> Assignee: Clemens Valiente
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L394
> Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions is created
> from repartitionTopicMetadata
> THEN we do ensureCoPartitioning on repartitionTopicMetadata
> THEN we create topics and partitions according to repartitionTopicMetadata
> THEN we use allRepartitionTopicPartitions to create our Cluster fullMetadata
> THEN we use fullMetadata to assign the tasks and no longer use
> repartitionTopicMetadata
> This results in any change to repartitionTopicMetadata in
> ensureCoPartitioning to be used for creating partitions but no tasks are ever
> created for any partition added by ensureCoPartitioning()
> the fix is easy: First ensureCoPartitioning() on repartitionTopicMetadata
> before creating allRepartitionTopicPartitions
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
