[
https://issues.apache.org/jira/browse/KAFKA-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Divij Vaidya updated KAFKA-14418:
---------------------------------
Description:
Today a user can change the number of partitions of
{{{}_{}}}{_}{{_consumer}}{_}{{{}_{}}}{{{}offsets{}}} topic by changing the
configuration value for {{offsets.topic.num.partitions}} or manually by using
CreatePartition API.
Changing offsets of this reserved partition leads to problems with consumer
groups unless you restart all brokers. Thus, there is a high probability that
is this operations is not done right, users may shoot themselves in the foot.
Example scenario:
[https://stackoverflow.com/questions/73944561/kafka-consumer-group-coordinator-inconsistent]
To remedy this, I propose the following changes:
1. `kafka-topic.sh` should explicitly block adding new partitions to internal
topics.
2. Add an operational guide to the docs for safely modifying partitions of
__consumer_offsets at [https://kafka.apache.org/documentation.html#basic_ops]
3. when {{offsets.topic.num.partitions }}is modified, it should update the
value of groupPartitionCount at
[https://github.com/apache/kafka/blob/3.3.1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L192]
without a broker startup
was:
Today a user can change the number of partitions of
{{{}__{}}}{{{}_consumer__{}}}{{{}offsets{}}} topic by changing the
configuration value for {{offsets.topic.num.partitions}} or manually by using
CreatePartition API.
Changing offsets of this reserved partition leads to problems with consumer
groups unless you restart all brokers. Thus, there is a high probability that
is this operations is not done right, users may shoot themselves in the foot.
Example scenario:
[https://stackoverflow.com/questions/73944561/kafka-consumer-group-coordinator-inconsistent]
To remedy this, I propose the following changes:
1. `kafka-topic.sh` should explicitly block adding new partitions to internal
topics
2. Add an operational guide to the docs for safely modifying partitions of
__consumer_offsets at [https://kafka.apache.org/documentation.html#basic_ops]
> Add safety checks for modifying partitions of __consumer_offsets
> ----------------------------------------------------------------
>
> Key: KAFKA-14418
> URL: https://issues.apache.org/jira/browse/KAFKA-14418
> Project: Kafka
> Issue Type: Improvement
> Reporter: Divij Vaidya
> Priority: Major
> Fix For: 3.4.0
>
>
> Today a user can change the number of partitions of
> {{{}_{}}}{_}{{_consumer}}{_}{{{}_{}}}{{{}offsets{}}} topic by changing the
> configuration value for {{offsets.topic.num.partitions}} or manually by using
> CreatePartition API.
> Changing offsets of this reserved partition leads to problems with consumer
> groups unless you restart all brokers. Thus, there is a high probability
> that is this operations is not done right, users may shoot themselves in the
> foot. Example scenario:
> [https://stackoverflow.com/questions/73944561/kafka-consumer-group-coordinator-inconsistent]
>
> To remedy this, I propose the following changes:
> 1. `kafka-topic.sh` should explicitly block adding new partitions to internal
> topics.
> 2. Add an operational guide to the docs for safely modifying partitions of
> __consumer_offsets at [https://kafka.apache.org/documentation.html#basic_ops]
> 3. when {{offsets.topic.num.partitions }}is modified, it should update the
> value of groupPartitionCount at
> [https://github.com/apache/kafka/blob/3.3.1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L192]
> without a broker startup
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
