You would add new brokers to the cluster, and then do a partition
reassignment to move some partitions to the new broker.
In the simplest example:
Say you have 1 topic with 3 partitions.
partition 0: brokers: 1,2
partition 1: brokers: 2,3
partition 2: brokers: 3,1
If you added 3 more brokers, you could do a reassignment so it is:
partition 0: brokers: 1,2
partition 1: brokers: 3,4
partition 2: brokers: 5,6
Which could be done with the following partition reassignment json:
{"partitions":[
{"topic": "test", "partition": 0, "replicas": [1,2] },
{"topic": "test", "partition": 1, "replicas": [3,4] },
{"topic": "test", "partition": 2, "replicas": [5,6] }
], "version":1
}
/usr/local/bin/kafka-reassign-partitions --execute --reassignment-json-file
reassignment.json --throttle 80000000
and remember to verify:
/usr/local/bin/kafka-reassign-partitions --verify --reassignment-json-file
reassignment.json
In reality you'd have multiple topics and probably many partitions, but the
principle is the same.
Documentation here:
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-4.ReassignPartitionsTool
On Fri, Dec 15, 2017 at 12:12 PM, ajay chaudhary <
juniora...@yahoo.com.invalid> wrote:
>
> Adding Users list to this email for help on below queries. please help us.
> Regards,Ajay chaudhary
>
> On Thursday 14 December 2017, 1:07:56 PM IST, ajay chaudhary <
> juniora...@yahoo.com> wrote:
>
> Hi Team,
> This is Ajay working with Yodlee India. We are trying to setup Kafka
> cluster for streaming and we need some clarifications.
> Could you please help us to understand this behavior.
> How do we add a capacity to existing Kafka cluster. Let us assume, we have
> Kafka cluster with 3 brokers and all brokers has single mount point
> allocated(log.dirs). we create a topic with some partitions and after some
> days mount point becomes 100% full may be due to unexpected data growth.
> how do we add space to Kafka cluster so that existing partitions (say
> partition 0 where it originally resides is full) make use of newly
> provisioned space?
> how would new messages be handled coming to this partition? Please help us
> understand this?
> Regards,Ajay chaudhary
>
