[
https://issues.apache.org/jira/browse/KAFKA-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286934#comment-16286934
]
ASF GitHub Bot commented on KAFKA-6150:
---------------------------------------
GitHub user guozhangwang opened a pull request:
https://github.com/apache/kafka/pull/4315
KAFKA-6150: KIP-204 part III; Change repartition topic segment size and ms
1. Create default internal topic configs in StreamsConfig, especially for
repartition topics change the segment size and time to smaller value.
2. Consolidate the default internal topic settings to InternalTopicManager
and simplify InternalTopicConfig correspondingly.
3. Add an integration test for purging data.
4. MINOR: change TopologyBuilderException to IllegalStateException in
StreamPartitionAssignor (part of
https://issues.apache.org/jira/browse/KAFKA-5660).
Here are a few public facing APIs that get added:
1. AbstractConfig#originalsWithPrefix(String prefix, boolean strip).
2. KafkaStreams constructor with Time object for convienent mocking in
tests.
Will update KIP-204 accordingly if people re-votes these changes.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guozhangwang/kafka K6150-segment-size
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/4315.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4315
----
commit 7f394817cc0a7a5606ee2cc1bae0d3e271c307b9
Author: Matthias J. Sax <[email protected]>
Date: 2017-11-21T02:17:57Z
KAFKA-4857: Replace StreamsKafkaClient with AdminClient in Kafka Streams
commit 49e832c297b93c432cdbffddca5635dbdecfc565
Author: Matthias J. Sax <[email protected]>
Date: 2017-12-04T19:36:29Z
Github comments
- fixed parameter passing for `retries`
- added couple of tests
commit ad7ccde56910a0b52825470fdd6a54339358445d
Author: Matthias J. Sax <[email protected]>
Date: 2017-12-04T21:01:46Z
rebased
commit bfd3937666ecc82e7584083fd1f17d0bacba8962
Author: Matthias J. Sax <[email protected]>
Date: 2017-12-05T05:38:36Z
fixed system test error message
commit 81b17ba0b60e9d11cd6a23d4f91786d499869393
Author: Guozhang Wang <[email protected]>
Date: 2017-12-07T02:53:33Z
Merge branch 'kafka-4857-admit-client' of https://github.com/mjsax/kafka
into K6150-segment-size
commit 3c286b94e2d698ab41c90668b2b9ada8212ab2d8
Author: Guozhang Wang <[email protected]>
Date: 2017-12-08T00:14:04Z
add internal topic types
commit 6c223b89c0b81769930c86390de7a8f745e71780
Author: Guozhang Wang <[email protected]>
Date: 2017-12-08T02:30:12Z
use default props for internal topics
commit 177d11a400f1574f7c1fbac1e233493e3d43b944
Author: Guozhang Wang <[email protected]>
Date: 2017-12-11T17:36:11Z
Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into
K6150-segment-size
commit ef57e80633bd39335bc184bed59d55179d18b69b
Author: Guozhang Wang <[email protected]>
Date: 2017-12-12T01:20:55Z
add integration test for purging repartition topic
----
> Make Repartition Topics Transient
> ---------------------------------
>
> Key: KAFKA-6150
> URL: https://issues.apache.org/jira/browse/KAFKA-6150
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Guozhang Wang
> Labels: operability
> Fix For: 1.1.0
>
>
> Unlike changelog topics, the repartition topics could just be short-lived.
> Today users have different ways to configure them with short retention such
> as enforce a short retention period or use AppendTime for repartition topics.
> All these would be cumbersome and Streams should just do this for the users.
> One way to do it is use the “purgeData” admin API (KIP-107) such that after
> the offset of the input topics are committed, if the input topics are
> actually repartition topics, we would purge the data immediately.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
