[
https://issues.apache.org/jira/browse/KAFKA-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901351#comment-16901351
]
ASF GitHub Bot commented on KAFKA-8611:
---------------------------------------
lkokhreidze commented on pull request #7170: KAFKA-8611 / Add
KStream#repartition operation
URL: https://github.com/apache/kafka/pull/7170
# KIP-221: Enhance DSL with Connecting Topic Creation and Repartition Hint
## Description
This is first PR for
[KIP-221](https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint).
Goal of this PR is to introduce new `KStream#repartition` operator that can be
used to for triggering repartitioning on `KStream` instance.
## Notable Changes
- Introduced
`org.apache.kafka.streams.processor.internals.InternalTopicProperties` class
that can be used for capturing repartition topic configurations passed via DSL
operations
- Enhanced `OptimizableRepartitionNode` with `StreamPartitioner<K,V>
partitioner` and `InternalTopicProperties internalTopicProperties` fields
- Added
`org.apache.kafka.streams.processor.internals.InternalTopologyBuilder#internalTopicNamesWithProperties`
map for storing mapping between internal topics and their corresponding
configuration. If configuration is present `RepartitionTopicConfig` is enriched
with configurations passed via DSL operations (In this case via
`org.apache.kafka.streams.kstream.Repartitioned` class).
- Added `KStreamRepartitionIntegrationTest` for testing different scenarios
of `KStream#repartition`
- - Repartition topic shouldn't be created when key changing operation
wasn't performed and `Repartitioned#numberOfPartitions` was not specified
- - Repartition topic should be created when key changing operation was
performed
- - Repartition topic should be created when key changing operation wasn't
performed, but `Repartitioned#numberOfPartitions` was specified
- - Repartition topic name should be picked up from `Repartitioned#name`
configuration
- - Repartition topic name should be generated automatically if
`Repartitioned#name` is not specified.
- - `KStream#repartition(KeyValueMapper, Repartitioned)` works as expected
with `KStream#groupByKey` (There should be only one repartition topic).
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add KStream#repartition operation
> ---------------------------------
>
> Key: KAFKA-8611
> URL: https://issues.apache.org/jira/browse/KAFKA-8611
> Project: Kafka
> Issue Type: New Feature
> Components: streams
> Reporter: Levani Kokhreidze
> Assignee: Levani Kokhreidze
> Priority: Minor
> Labels: kip
>
> When using DSL in Kafka Streams, data re-partition happens only when
> key-changing operation is followed by stateful operation. On the other hand,
> in DSL, stateful computation can happen using _transform()_ operation as
> well. Problem with this approach is that, even if any upstream operation was
> key-changing before calling _transform()_, no auto-repartition is triggered.
> If repartitioning is required, a call to _through(String)_ should be
> performed before _transform()_. With the current implementation, burden of
> managing and creating the topic falls on user and introduces extra complexity
> of managing Kafka Streams application.
> KIP-221:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
