You need to do a partition reassignment to increase or decrease the replication factor. It's tediously manual, but it's just json so it's trivial to manipulate which is probably why it's still tediously manual.
There's a guide here although it's ageing a little: http://www.allprogrammingtutorials.com/tutorials/changing-replication-factor-of-topic-in-kafka.php and here: https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor You can use some built in tooling to help: Create a topics.json file: {"topics": [{"topic": "__consumer_offsets"}], "version":1 } Assuming you currently have brokers 1,2,3,4 and you want to end up with 1,2: bin/kafka-reassign-partitions.sh --zookeeper your_zk_string --generate --topics-to-move-json-file topics.json --generate --broker-list 1,2,3 >From that you want to remove broker 3 from the reassignment proposal, turning the 3 element array into 2. If you're on a newer version of kafka you also get log_dirs array in the output. For me that's always "any" so I just take it out. If they're different for you you'll have more work to do. If you don't want to edit it manually you could use some jq: bin/kafka-reassign-partitions.sh --zookeeper your_zk_string --generate --topics-to-move-json-file topics.json --generate --broker-list 1,2,3 | sed -e '1,/Proposed/d' | jq 'del(.partitions[].replicas[] | select(. == 3)) | del(.partitions[].log_dirs)' > reassignment.json If you're happy with the reassignment file then execute it, running with --verify instead of --execute to see when it's done. bin/kafka-reassign-partitions.sh --zookeeper your_zk_string --reassignment-json-file reassignment.json --execute And then double check with: bin/kafka-topics.sh --zookeeper your_zk_string --describe --topic __consumer_offsets On Tue, Sep 18, 2018 at 9:34 AM Dylan Martin <dmar...@istreamplanet.com> wrote: > I have a cluster with 4 brokers and I want to reduce it to 2 brokers. I > cannot re-assign __consumer_offsets because it wants at least 3 brokers. > > > Is there a way to do this? Or am I going to have to trash my cluster and > start over? > > > -Dylan > > > (206) 855-9740 - Home > > (206) 235-8809 - Cell > > The information contained in this email message, and any attachment > thereto, is confidential and may not be disclosed without the sender's > express permission. If you are not the intended recipient or an employee or > agent responsible for delivering this message to the intended recipient, > you are hereby notified that you have received this message in error and > that any review, dissemination, distribution or copying of this message, or > any attachment thereto, in whole or in part, is strictly prohibited. If you > have received this message in error, please immediately notify the sender > by telephone, fax or email and delete the message and all of its > attachments. Thank you. >
