Since you have replication factor 3 and only 3 brokers, you can't move data around in the existing cluster since a partition can have at most replica on each broker. You will need to add the new brokers in first before running the reassignment tool.
Thanks, Jun On Tue, Jul 8, 2014 at 10:26 AM, Florian Dambrine <flor...@gumgum.com> wrote: > I let the tool running for an entire weekend on the test cluster and on > Monday it was still saying "failed"... > > I have 500 Go per Kafka node and it is a 8 nodes cluster. > > I am also wondering if I am using the tool correctly. Currently I am > running the tool to rebalance everything across the entire cluster. As I > have 3 replicas the tool requires at least 3 brokers. > > Should I add 3 new Kafka nodes and rebalance some topics to these new nodes > only? I am afraid to unbalance the cluster with this option. > > Any suggestions? > > Thanks for your help. > > > On Mon, Jul 7, 2014 at 9:29 PM, Jun Rao <jun...@gmail.com> wrote: > > > The failure could mean that the reassignment is still in progress. If you > > have lots of data, it may take some time to move the data to new brokers. > > You could observe the max lag in each broker to see how far behind new > > replicas are (see http://kafka.apache.org/documentation.html#monitoring > ). > > > > Thanks, > > > > Jun > > > > > > On Mon, Jul 7, 2014 at 4:42 PM, Florian Dambrine <flor...@gumgum.com> > > wrote: > > > > > When I run the tool with the --verify option it says failed for the > some > > > partitions. > > > > > > The problem is I do not know if it is a zookeeper issue or if the tool > > > really failed. > > > > > > I faced one time the zookeeper issue ( > > > https://issues.apache.org/jira/browse/KAFKA-1382) and by killing the > > > responsible Kafka the partition switched from failed to completed > > > successfully. > > > > > > What should I do when the Kafka tool says that it failed to move the > > > partition? > > > > > > > > > > > > > > > On Mon, Jul 7, 2014 at 4:33 PM, Clark Haskins > > > <chask...@linkedin.com.invalid > > > > wrote: > > > > > > > How does it get stuck? > > > > > > > > -Clark > > > > > > > > Clark Elliott Haskins III > > > > LinkedIn DDS Site Reliability Engineer > > > > Kafka, Zookeeper, Samza SRE > > > > Mobile: 505.385.1484 > > > > BlueJeans: https://www.bluejeans.com/chaskins > > > > > > > > > > > > chask...@linkedin.com > > > > https://www.linkedin.com/in/clarkhaskins > > > > There is no place like 127.0.0.1 > > > > > > > > > > > > > > > > > > > > On 7/7/14, 3:49 PM, "Florian Dambrine" <flor...@gumgum.com> wrote: > > > > > > > > >Hi, > > > > > > > > > >I am trying to add new brokers to an existing 8 nodes Kafka cluster. > > We > > > > >have around 10 topics and the number of partition is set to 50. In > > order > > > > >to > > > > >test the reassgin-partitions scripts, I tried on a sandbox cluster > the > > > > >following steps. > > > > > > > > > >I developed a script which is able to parse the reassignment > partition > > > > >plan > > > > >given by the Kafka tool in smaller pieces (reassigning maximum 10 > > > > >partitions at a time). > > > > > > > > > >Unfortunately I faced some issues with the tool that sometimes get > > stuck > > > > >on > > > > >one partition. In this case I have to kill and restart the three > > Kafkas > > > on > > > > >which the partition has been relocated to unlock the process (One > > kafka > > > at > > > > >a time). > > > > > > > > > >Moreover, I have also faced these two issues that are already on > Jira: > > > > > > > > > >https://issues.apache.org/jira/browse/KAFKA-1382 > > > > >https://issues.apache.org/jira/browse/KAFKA-1479 > > > > > > > > > >We really need to add new nodes to our Kafka cluster, does anybody > > have > > > > >already rebalance a Kafka 0.8.1.1? What could you advise me? > > > > > > > > > >Thanks, and feel free to ask me if you need more details. > > > > > > > > > > > > > > > > > > > >-- > > > > >*Florian Dambrine* | Intern, Big Data > > > > >*GumGum* <http://www.gumgum.com/> | *Ads that stick* > > > > >209-797-3994 | flor...@gumgum.com > > > > > > > > > > > > > > > > > -- > > > *Florian Dambrine* | Intern, Big Data > > > *GumGum* <http://www.gumgum.com/> | *Ads that stick* > > > 209-797-3994 | flor...@gumgum.com > > > > > > > > > -- > *Florian Dambrine* | Intern, Big Data > *GumGum* <http://www.gumgum.com/> | *Ads that stick* > 209-797-3994 | flor...@gumgum.com >
