Can you copy/paste the json you are passing to the reassignment tool? Plus the commands. Also do a describe on your topics.
-Clark Clark Elliott Haskins III LinkedIn DDS Site Reliability Engineer Kafka, Zookeeper, Samza SRE Mobile: 505.385.1484 BlueJeans: https://www.bluejeans.com/chaskins chask...@linkedin.com https://www.linkedin.com/in/clarkhaskins There is no place like 127.0.0.1 On 7/8/14, 10:26 AM, "Florian Dambrine" <flor...@gumgum.com> wrote: >I let the tool running for an entire weekend on the test cluster and on >Monday it was still saying "failed"... > >I have 500 Go per Kafka node and it is a 8 nodes cluster. > >I am also wondering if I am using the tool correctly. Currently I am >running the tool to rebalance everything across the entire cluster. As I >have 3 replicas the tool requires at least 3 brokers. > >Should I add 3 new Kafka nodes and rebalance some topics to these new >nodes >only? I am afraid to unbalance the cluster with this option. > >Any suggestions? > >Thanks for your help. > > >On Mon, Jul 7, 2014 at 9:29 PM, Jun Rao <jun...@gmail.com> wrote: > >> The failure could mean that the reassignment is still in progress. If >>you >> have lots of data, it may take some time to move the data to new >>brokers. >> You could observe the max lag in each broker to see how far behind new >> replicas are (see >>http://kafka.apache.org/documentation.html#monitoring). >> >> Thanks, >> >> Jun >> >> >> On Mon, Jul 7, 2014 at 4:42 PM, Florian Dambrine <flor...@gumgum.com> >> wrote: >> >> > When I run the tool with the --verify option it says failed for the >>some >> > partitions. >> > >> > The problem is I do not know if it is a zookeeper issue or if the tool >> > really failed. >> > >> > I faced one time the zookeeper issue ( >> > https://issues.apache.org/jira/browse/KAFKA-1382) and by killing the >> > responsible Kafka the partition switched from failed to completed >> > successfully. >> > >> > What should I do when the Kafka tool says that it failed to move the >> > partition? >> > >> > >> > >> > >> > On Mon, Jul 7, 2014 at 4:33 PM, Clark Haskins >> > <chask...@linkedin.com.invalid >> > > wrote: >> > >> > > How does it get stuck? >> > > >> > > -Clark >> > > >> > > Clark Elliott Haskins III >> > > LinkedIn DDS Site Reliability Engineer >> > > Kafka, Zookeeper, Samza SRE >> > > Mobile: 505.385.1484 >> > > BlueJeans: https://www.bluejeans.com/chaskins >> > > >> > > >> > > chask...@linkedin.com >> > > https://www.linkedin.com/in/clarkhaskins >> > > There is no place like 127.0.0.1 >> > > >> > > >> > > >> > > >> > > On 7/7/14, 3:49 PM, "Florian Dambrine" <flor...@gumgum.com> wrote: >> > > >> > > >Hi, >> > > > >> > > >I am trying to add new brokers to an existing 8 nodes Kafka >>cluster. >> We >> > > >have around 10 topics and the number of partition is set to 50. In >> order >> > > >to >> > > >test the reassgin-partitions scripts, I tried on a sandbox cluster >>the >> > > >following steps. >> > > > >> > > >I developed a script which is able to parse the reassignment >>partition >> > > >plan >> > > >given by the Kafka tool in smaller pieces (reassigning maximum 10 >> > > >partitions at a time). >> > > > >> > > >Unfortunately I faced some issues with the tool that sometimes get >> stuck >> > > >on >> > > >one partition. In this case I have to kill and restart the three >> Kafkas >> > on >> > > >which the partition has been relocated to unlock the process (One >> kafka >> > at >> > > >a time). >> > > > >> > > >Moreover, I have also faced these two issues that are already on >>Jira: >> > > > >> > > >https://issues.apache.org/jira/browse/KAFKA-1382 >> > > >https://issues.apache.org/jira/browse/KAFKA-1479 >> > > > >> > > >We really need to add new nodes to our Kafka cluster, does anybody >> have >> > > >already rebalance a Kafka 0.8.1.1? What could you advise me? >> > > > >> > > >Thanks, and feel free to ask me if you need more details. >> > > > >> > > > >> > > > >> > > >-- >> > > >*Florian Dambrine* | Intern, Big Data >> > > >*GumGum* <http://www.gumgum.com/> | *Ads that stick* >> > > >209-797-3994 | flor...@gumgum.com >> > > >> > > >> > >> > >> > -- >> > *Florian Dambrine* | Intern, Big Data >> > *GumGum* <http://www.gumgum.com/> | *Ads that stick* >> > 209-797-3994 | flor...@gumgum.com >> > >> > > > >-- >*Florian Dambrine* | Intern, Big Data >*GumGum* <http://www.gumgum.com/> | *Ads that stick* >209-797-3994 | flor...@gumgum.com
