GitHub user ymatsuda opened a pull request:
https://github.com/apache/kafka/pull/353
KAFKA-2652: integrate new group protocol into partition grouping
@guozhangwang
* added ```PartitionGrouper``` (abstract class)
* This class is responsible for grouping partitions to form tasks.
* Users may implement this class for custom grouping.
* added ```DefaultPartitionGrouper```
* our default implementation of ```PartitionGrouper```
* added ```KafkaStreamingPartitionAssignor```
* We always use this as ```PartitionAssignor``` of stream consumers.
* Actual grouping is delegated to ```PartitionGrouper```.
* ```TopologyBuilder```
* added ```topicGroups()```
* This returns groups of related topics according to the topology
* added ```copartitionSources(sourceNodes...)```
* This is used by DSL layer. It asserts the specified source nodes must
be copartitioned.
* added ```copartitionGroups()``` which returns groups of copartitioned
topics
* KStream layer
* keep track of source nodes to determine copartition sources when steams
are joined
* source nodes are set to null when partitioning property is not preserved
(ex. ```map()```, ```transform()```), and this indicates the stream is no
longer joinable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ymatsuda/kafka grouping
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/353.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #353
----
commit 708718c1be23fad25fa6206f665cbb619c1b5097
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-19T19:38:06Z
partition grouping
commit d2bae046b5509022e2821a2c5eb08853d228e791
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-19T20:19:54Z
wip
commit 86fa8110b23ee1992fbd19daa08c63a4b427448e
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-20T20:01:37Z
long task id
commit 4f4f9ac642ebe0eae33a5c8464309106e9239f2e
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-20T20:03:15Z
Merge branch 'trunk' of github.com:apache/kafka into grouping
commit e4ecf39b9ab0b0f4c915a4f43cfe771b1de69f7f
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-21T19:33:05Z
joinability
commit 37d72a691173a8fe878ac3d99e8973e72f5675c6
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-21T19:33:48Z
Merge branch 'trunk' of github.com:apache/kafka into grouping
commit f68723bab83c3a3f1c15872f4f24bc932df8198f
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-22T18:21:31Z
partition assignor
commit 457cf270222139eae89750781d09abaa07120932
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-22T18:21:40Z
Merge branch 'trunk' of github.com:apache/kafka into grouping
commit 13f3ad703960581229d511287f27345c567b5d3e
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-22T18:34:52Z
complete undoing long taskid
commit 98f3bcc1896fd159ccbbd37fc65b1d9d6f568bb9
Author: Yasuhiro Matsuda <[email protected]>
Date: 2015-10-22T18:45:38Z
fix a test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
