Hi,
with Kafka 0.8 it is possible to add new partitions on newly added brokers
and supply a partition assignment to put the new partitions mainly on the
new brokers (4 and 5 are the new brokers):
bin/kafka-add-partitions.sh --topic scale-test-001 \
--partition 14 \
--replica-assignment-list
4:5,4:1,4:2,4:3,4:5,4:1,4:2,5:3,5:4,5:1,5:2,5:3,5:4,5:1 \
--zookeeper 127.0.0.1:2181
For 0.8.1+ the kafka-add-partitions.sh tool was merged into
kafka-topics.sh, but when you try to execute something similar you receive
the following error (in Kafka 0.8.2.1):
kafka_2.10-0.8.2.1$ bin/kafka-topics.sh --alter --topic scale-test-002 \
> --zookeeper 127.0.0.1:2181 \
> --partitions 35 \
> --replica-assignment
2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,4:5,4:2,4:2,4:3,4:5,4:3,4:2,5:3,5:4,5:4,5:2,5:3,5:4,5:3
Option "[replica-assignment]" can't be used with option"[partitions]"
However, upon investigation of alterTopics in TopicCommand.scala the code
it wants to execute is:
val nPartitions = opts.options.valueOf(opts.partitionsOpt).intValue
val replicaAssignmentStr = opts.options.valueOf(opts.replicaAssignmentOpt)
AdminUtils.addPartitions(zkClient, topic, nPartitions,
replicaAssignmentStr, config = configs)
So assigning both the --partitions and the --replica-assignment parameters
should be totally fine.
The issue is with the following line in checkArgs:
CommandLineUtils.checkInvalidArgs(parser, options, replicaAssignmentOpt,
allTopicLevelOpts -- Set(alterOpt, createOpt) + partitionsOpt +
replicationFactorOpt)
If it is removed, then the above command executes just fine. The created
partitions are as well filled quite happily.
I'm not fully sure what the correct configuration of the
replicaAssignmentOpt should be, so I don't provide a patch, but it would be
great if that could be fixed.
Best regards,
Stefan
--
*Dr. Stefan Schadwinkel*
Senior Big Data Developer
stefan.schadwin...@smaato.com
Smaato Inc.
San Francisco – New York - Hamburg - Singapore
www.smaato.com
Germany:
Valentinskamp 70, Emporio, 19th Floor
20355 Hamburg
T +49 (40) 3480 949 0
F +49 (40) 492 19 055
The information contained in this communication may be CONFIDENTIAL and is
intended only for the use of the recipient(s) named above. If you are not
the intended recipient, you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of its contents, is
strictly prohibited. If you have received this communication in error,
please notify the sender and delete/destroy the original message and any
copy of it from your computer or paper files.
