[
https://issues.apache.org/jira/browse/KAFKA-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298661#comment-16298661
]
ASF GitHub Bot commented on KAFKA-6391:
---------------------------------------
GitHub user cvaliente opened a pull request:
https://github.com/apache/kafka/pull/4347
KAFKA-6391 ensure topics are created with correct partitions BEFORE
building the…
ensure topics are created with correct partitions BEFORE building the
metadata for our stream tasks
First ensureCoPartitioning() on repartitionTopicMetadata before creating
allRepartitionTopicPartitions
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cvaliente/kafka KAFKA-6391
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/4347.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4347
----
commit bda1803d50d984ef4860579d508c37487df9781a
Author: Clemens Valiente <clemens.valiente@...>
Date: 2017-12-20T15:45:41Z
ensure topics are created with correct partitions BEFORE building the
metadata for our stream tasks
----
> output from ensure copartitioning is not used for Cluster metadata, resulting
> in partitions without tasks working on them
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-6391
> URL: https://issues.apache.org/jira/browse/KAFKA-6391
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 1.0.0
> Reporter: Clemens Valiente
> Assignee: Clemens Valiente
> Original Estimate: 20m
> Remaining Estimate: 20m
>
> https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L394
> Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions is created
> from repartitionTopicMetadata
> THEN we do ensureCoPartitioning on repartitionTopicMetadata
> THEN we create topics and partitions according to repartitionTopicMetadata
> THEN we use allRepartitionTopicPartitions to create our Cluster fullMetadata
> THEN we use fullMetadata to assign the tasks and no longer use
> repartitionTopicMetadata
> This results in any change to repartitionTopicMetadata in
> ensureCoPartitioning to be used for creating partitions but no tasks are ever
> created for any partition added by ensureCoPartitioning()
> the fix is easy: First ensureCoPartitioning() on repartitionTopicMetadata
> before creating allRepartitionTopicPartitions
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
