Re: Topic partition reassignment fails because of non-monotonic offsets (Kafka 2.6)

2020-10-30 Thread Joe Ammann
We were ultimately able to solve this issue - mainly by sitting and waiting The issue was indeed that somewhen, somehow, the data on the leader of this __consumer_offset-18 partition got corrupted. This probably happened during the upgrade from Kafka 2.2 -> 2.6. We were doing this in a rather dang

Topic partition reassignment fails because of non-monotonic offsets (Kafka 2.6)

2020-10-26 Thread Joe Ammann
We did an upgrade from Kafka 2.2 to 2.6, followed by a migration (through reassign-partitions) from old to new brokers. As described in https://stackoverflow.com/questions/64514851/apache-kafka-kafka-common-offsetsoutoforderexception-when-reassigning-consume, all but 1 partition (__consumer_offset

RE: will partition reassignment cause same message being processed in parallel?

2020-08-14 Thread Jin, Yun
. In this case, the same message may also be processed in parallel. -Original Message- From: Yingshuan Song Sent: Thursday, August 13, 2020 23:42 To: users@kafka.apache.org Subject: Re: will partition reassignment cause same message being processed in parallel? Yes,it is possible. Think

Re: will partition reassignment cause same message being processed in parallel?

2020-08-13 Thread Yingshuan Song
; > Partition may be reassigned if the consumer is not reachable. Will > partition reassignment cause same message being processed in parallel? > Suppose if Kafka found consumer A is not reachable (maybe because of > network problem), it assigns the partition to consumer B. But actually

will partition reassignment cause same message being processed in parallel?

2020-08-12 Thread Jin, Yun
Hi, Partition may be reassigned if the consumer is not reachable. Will partition reassignment cause same message being processed in parallel? Suppose if Kafka found consumer A is not reachable (maybe because of network problem), it assigns the partition to consumer B. But actually consumer A is

Re: Partition Reassignment is getting stuck

2019-11-14 Thread Ashutosh singh
ineLogDirectoryCount" or use kafka-log-dirs.sh > > to understand the issue with that directory. In most cases, it would be > the > > a KafkaStorage Exception. > > The partition reassignment would also be stuck/waiting because of this, > > when the reassignment j

Re: Partition Reassignment is getting stuck

2019-11-13 Thread Liam Clarke
be on a offline directory. > Look into the metric "offlineLogDirectoryCount" or use kafka-log-dirs.sh > to understand the issue with that directory. In most cases, it would be the > a KafkaStorage Exception. > The partition reassignment would also be stuck/waiting because of this

RE: Partition Reassignment is getting stuck

2019-11-13 Thread Koushik Chitta
The topic partition having the ISR issue might be on a offline directory. Look into the metric "offlineLogDirectoryCount" or use kafka-log-dirs.sh to understand the issue with that directory. In most cases, it would be the a KafkaStorage Exception. The partition reassignment wou

Re: Partition Reassignment is getting stuck

2019-11-13 Thread M. Manna
On Wed, 13 Nov 2019 at 13:10, Ashutosh singh wrote: > Yeah, Although it wouldn't have any impact but I will have to try this > tonight as it is peak business hours now. > Instead deleting all data I will try to delete topic partitions which are > having issues and then restart the broker. I bel

Re: Partition Reassignment is getting stuck

2019-11-13 Thread Ashutosh singh
Yeah, Although it wouldn't have any impact but I will have to try this tonight as it is peak business hours now. Instead deleting all data I will try to delete topic partitions which are having issues and then restart the broker. I believe it should catch up but I will let you know. On Wed, No

Re: Partition Reassignment is getting stuck

2019-11-13 Thread M. Manna
On Wed, 13 Nov 2019 at 12:41, Ashutosh singh wrote: > Hi, > > All of a sudden I see under replicated partition in our Kafka cluster and > it is not getting replicated. It seems it is getting stuck somewhere. In > sync replica is missing only form one of the broker it seems there is some > issue

Partition Reassignment is getting stuck

2019-11-13 Thread Ashutosh singh
Hi, All of a sudden I see under replicated partition in our Kafka cluster and it is not getting replicated. It seems it is getting stuck somewhere. In sync replica is missing only form one of the broker it seems there is some issue with that broker but other hand there are many others topic on t

Re: Cancel partition reassignment?

2018-02-08 Thread Ted Yu
Have you seen this thread ? http://search-hadoop.com/m/Kafka/uyzND1pHiNuYt8hc1?subj=Re+Question+Kafka+Reassign+partitions+tool On Thu, Feb 8, 2018 at 4:12 PM, Dylan Martin wrote: > Hi all. > > > I'm trying to cancel a failed partition reassignment. I've heard that

Cancel partition reassignment?

2018-02-08 Thread Dylan Martin
Hi all. I'm trying to cancel a failed partition reassignment. I've heard that this can be done by deleting /admin/reassign_partitions in zookeeper. I've tried and /admin/reassign_partitions won't go away. Does anyone know a way to cancel a partition reassignment? -Dy

Re: Partition reassignment data file is empty

2018-01-01 Thread Ted Yu
xplain what the error means? The json > is > > > not > > > > empty > > > > > > > > $ cat increase-replication-factor.json > > > > {"version":1, > > > > "partitions":[ > > > > {"topic&quo

Re: Partition reassignment data file is empty

2017-12-31 Thread allen chan
> $ cat increase-replication-factor.json > > > {"version":1, > > > "partitions":[ > > > {"topic":"metrics","partition":0,"replicas":[1,2]}, > > > {"topic":"metrics&

Re: Partition reassignment data file is empty

2017-12-31 Thread Ted Yu
[ > > {"topic":"metrics","partition":0,"replicas":[1,2]}, > > {"topic":"metrics","partition":1,"replicas":[2,3]}, > > ]} > > > > $ sudo /opt/kafka/kafka_2

Re: Partition reassignment data file is empty

2017-12-31 Thread Brett Rann
ot;topic":"metrics","partition":1,"replicas":[2,3]}, > ]} > > $ sudo /opt/kafka/kafka_2.12-0.11.0.1/bin/kafka-reassign-partitions.sh > --zookeeper server1:2181 --reassignment-json-file > increase-replication-factor.json --execute > Partitions r

Partition reassignment data file is empty

2017-12-30 Thread allen chan
json-file increase-replication-factor.json --execute Partitions reassignment failed due to Partition reassignment data file is empty kafka.common.AdminCommandFailedException: Partition reassignment data file is empty at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:

Lots of warns about LogCleaningPaused during partition reassignment

2017-11-28 Thread BGCH
Hello! We tried to migrate data from 0.10.2.1 cluster to 0.11.0.2. Firstly we spread topics to both clusters. There were lots of problems and restarts of some nodes of both clusters (we probably shouldn't do that). All this ended up with a state when we had lots of exceptions from 2 nodes of

How to debug why partition reassignment failure?

2017-05-30 Thread Kerry Wei
Hi, I'm using kafka-reassign-partitions.sh to move partitions around, however, sometimes I got partition reassignment failure. The cluster is healthy before the rebalance, and a retry after 10 mins resolved the problem. However, I wonder if there's a way I can check why the reassignment

Kafka Partition Reassignment service (based on an adoption marketplace model)

2016-05-10 Thread Soumyajit Sahu
Hello all, I have a design for a solution to the problem of "partition imbalances in Kafka clusters". It would be great to get some feedback on it. https://soumyajitsahu.wordpress.com/2016/05/11/kafka-partition-reassignment-service-using-an-adoption-marketplace-model/ I have also put

Safely cancel partition reassignment task

2015-08-19 Thread tao xiao
Hi team, What is the best way to cancel an in progress partition reassignment job? I know it saves the json in /admin/reassign_partition in zk. Is it ok to delete the znode?

Kafka partition reassignment causes replica number and Isr to go above configured value

2015-08-05 Thread Ye Hong
We recently performed a partition reassignment. Our Kafka cluster is on 0.8.1.1, and all topics are configured to have 2 replicas. After the completion of reassignment, we have noticed that some of the partitions has 3 replicas and 3 Isrs. for example: Topic: topic1Partition: 7Leader

Partition reassignment causes replica number and Isr to go above configured value

2015-08-05 Thread Ye Hong
Hi All, We recently performed a partition reassignment on one of our Kafka cluster. (0.8.1.1, all topics configured to have 2 replicas.) After the completion of reassignment, we have noticed that some of the partitions has 3 replicas and 3 Isrs instead of 2. for example: Topic: topic1

Re: Specify leader when doing partition reassignment

2015-08-05 Thread tao xiao
Thank you Todd for the detailed answer On Wed, Aug 5, 2015 at 9:09 PM Todd Palino wrote: > To make sure you have a complete answer here, the order of the replica list > that you specify in the partition reassignment will affect the leader > selection, but if the current leader is i

Re: Specify leader when doing partition reassignment

2015-08-05 Thread Todd Palino
To make sure you have a complete answer here, the order of the replica list that you specify in the partition reassignment will affect the leader selection, but if the current leader is in the new replica list, it will not change the leadership to change. That is, if your current replica list is

Re: Specify leader when doing partition reassignment

2015-08-05 Thread tao xiao
o wrote: > > > Hi team, > > > > Is it possible to specify a leader broker for each topic partition when > > doing partition reassignment? > > > > For example I have following json. Is the first broker in the replicas > list > > by default the leader of the

Re: Specify leader when doing partition reassignment

2015-08-05 Thread Jilin Xie
ach topic partition when > doing partition reassignment? > > For example I have following json. Is the first broker in the replicas list > by default the leader of the partition e.g. broker 3 is the leader of topic > test5 and broker 2 is the leader of topic test3. or does Kafka > au

Specify leader when doing partition reassignment

2015-08-05 Thread tao xiao
Hi team, Is it possible to specify a leader broker for each topic partition when doing partition reassignment? For example I have following json. Is the first broker in the replicas list by default the leader of the partition e.g. broker 3 is the leader of topic test5 and broker 2 is the leader

Re: cannot make another partition reassignment due to the previous partition reassignment failure

2015-06-15 Thread Yu Yang
nikumar > > > On Tue, Jun 16, 2015 at 8:15 AM, Yu Yang wrote: > > > HI, > > > > We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment > for > > some topic partitions in the cluster. Due to broker failure, the > partition > > reassignme

Re: cannot make another partition reassignment due to the previous partition reassignment failure

2015-06-15 Thread Manikumar Reddy
y I did a partition assignment for > some topic partitions in the cluster. Due to broker failure, the partition > reassignment failed. I cannot do another partition assignment now, and > always get errors as follows. How can we work around this? I have tried > google for answers, but

cannot make another partition reassignment due to the previous partition reassignment failure

2015-06-15 Thread Yu Yang
HI, We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment for some topic partitions in the cluster. Due to broker failure, the partition reassignment failed. I cannot do another partition assignment now, and always get errors as follows. How can we work around this? I have tried

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-18 Thread Chris Neal
ou! > >>>>>>> Chris > >>>>>>> > >>>>>>>> On Sun, May 17, 2015 at 12:58 PM, Clark Haskins > > >>>>>> wrote: > >>>>>>>> > >>>>>>>> Do a get /admin/reassign_partitions > &

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Clark Haskins
ay 17, 2015, at 10:20 AM, Chris Neal wrote: >>>>>>>>> >>>>>>>>> Sure thing :) >>>>>>>>> Hopefully I did this right. Somewhat of a Zookeeper noob. >>>>>>>>> >>>>>>>&

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Chris Neal
com <http://myhost.mydomain.com:2181/ > >:2181 > >>>> ls > >>>>>>> /admin/reassign_partitions > >>>>>>> Connecting to myhost.mydomain.com < > http://myhost.mydomain.com:2181/ > >>>>>>> :2181 >

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Clark Haskins
>>>>>>> :2181 >>>>>>> >>>>>>> WATCHER:: >>>>>>> >>>>>>> WatchedEvent state:SyncConnected type:None path:null >>>>>>> [] >>>>>>> >>>>&g

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Chris Neal
te:SyncConnected type:None path:null >>> >>> [] >>> >>> >>> >>> Hope that is helpful :) >>> >>> If this is not what you were asking for, please just let me know. >>> >>> Thank you! >>> >>> Chris &g

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Chris Neal
_partitions znode is the important one. Please paste the >> >>>> contents of it. That node should only exist while there is a >> >> reassignment >> >>>> in progress. >> >>>> >> >>>> You can probably fi

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Chris Neal
d only exist while there is a > >> reassignment > >>>> in progress. > >>>> > >>>> You can probably fix this up by forcing a new controller to come > online > >> by > >>>> deleting /controller > >>>> > >>>

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Clark Haskins
;>>> Sent from my iPhone >>>> >>>>> On May 17, 2015, at 10:14 AM, Chris Neal wrote: >>>>> >>>>> Hi Clark, >>>>> >>>>> Thank you for your reply! I do see that znode under /admin: >>>>> &g

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Chris Neal
; > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>> -server myhost.mydomain.com:2181 ls /admin > >>> Connecting to myhost.mydomain.com:2181 > >>> > >>> WATCHER:: > >>> > >>> WatchedEvent stat

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Clark Haskins
eper/bin/zkCli.sh >>> -server myhost.mydomain.com:2181 ls /admin >>> Connecting to myhost.mydomain.com:2181 >>> >>> WATCHER:: >>> >>> WatchedEvent state:SyncConnected type:None path:null >>> [reassign_partitions, delete_topics] &g

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Chris Neal
ure what this tells me though :) > > Again, thanks for your time. > > Chris > > > >> On Sun, May 17, 2015 at 12:20 AM, Clark Haskins > wrote: > >> > >> Does the partition reassignment znode exist under /admin in zookeeper? > >> > >> -Clark

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Clark Haskins
cting to myhost.mydomain.com:2181 > > WATCHER:: > > WatchedEvent state:SyncConnected type:None path:null > [reassign_partitions, delete_topics] > > I'm not sure what this tells me though :) > Again, thanks for your time. > Chris > >> On Sun, May 17, 2015

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-17 Thread Chris Neal
one path:null [reassign_partitions, delete_topics] I'm not sure what this tells me though :) Again, thanks for your time. Chris On Sun, May 17, 2015 at 12:20 AM, Clark Haskins wrote: > Does the partition reassignment znode exist under /admin in zookeeper? > > -Clark > > Sent from

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-16 Thread Clark Haskins
Does the partition reassignment znode exist under /admin in zookeeper? -Clark Sent from my iPhone > On May 16, 2015, at 7:16 PM, Chris Neal wrote: > > Sorry for bumping my own thread. :S Just wanted to get it in front of some > eyes again! > > Thanks for your time and help

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-16 Thread Chris Neal
, I get this: > > Partitions reassignment failed due to Partition reassignment currently in > progress for Map(). Aborting operation > kafka.common.AdminCommandFailedException: Partition reassignment currently > in progress for Map(). Aborting operation > at > kafka.admin.ReassignPar

Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-14 Thread Chris Neal
Hi All, I am running: kafka_2.10-0.8.1.1, and when I run the reassign-partitions.sh script, I get this: Partitions reassignment failed due to Partition reassignment currently in progress for Map(). Aborting operation kafka.common.AdminCommandFailedException: Partition reassignment currently in

Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
eds to be resolved, you might need to bounce both of the brokers >> who think itself as controller respectively. The new controller should >>be >> able to continue the partition reassignment. >> >> From: Wes Chow >> Reply-To: "users@kafka.apache.org

Re: partition reassignment stuck

2015-04-21 Thread Wesley Chow
angjie Qin" wrote: > Yes, should be broker 25 thread 0 from the log. > This needs to be resolved, you might need to bounce both of the brokers > who think itself as controller respectively. The new controller should be > able to continue the partition reassignment. > > F

Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
Yes, should be broker 25 thread 0 from the log. This needs to be resolved, you might need to bounce both of the brokers who think itself as controller respectively. The new controller should be able to continue the partition reassignment. From: Wes Chow mailto:w...@chartbeat.com>> Re

Re: partition reassignment stuck

2015-04-21 Thread Wes Chow
fka.apache.org>" mailto:users@kafka.apache.org>> Subject: Re: partition reassignment stuck Not for that particular partition, but I am seeing these errors on 28: kafka.common.NotAssignedReplicaException: Leader 28 failed to record follower 25's position 0 for partition [clic

Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>> Subject: Re: partition reassignment stuck Not for that particular partition, but I am seeing these errors on 28: kafka.common.NotAssignedReplicaException: Leader 28 failed to record follower 25's position 0 f

Re: partition reassignment stuck

2015-04-21 Thread Wes Chow
ated if new replicas is assigned to the broker. We might want to know what caused the UnknownException. Did you see any error log on broker 28? Jiangjie (Becket) Qin Wes Chow <mailto:w...@chartbeat.com> April 21, 2015 at 12:16 PM I started a partition reassignment (this is a 8.1.1 cluste

Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
Chow" wrote: >I started a partition reassignment (this is a 8.1.1 cluster) some time >ago and it seems to be stuck. Partitions are no longer getting moved >around, but it seems like the cluster is operational otherwise. The >stuck nodes have a lot of .index files

partition reassignment stuck

2015-04-21 Thread Wes Chow
I started a partition reassignment (this is a 8.1.1 cluster) some time ago and it seems to be stuck. Partitions are no longer getting moved around, but it seems like the cluster is operational otherwise. The stuck nodes have a lot of .index files, and their logs show errors

partition reassignment

2015-04-03 Thread Wes Chow
I'm in the process of reassigning partitions away from failing machines and it appears to be stuck. One thought is because our machines are failing at a very high rate and so some partitions no longer have any live replicas at all. At this point I don't care about the data, I just want to get

Re: Partition reassignment doesn't work on few partitions

2015-01-22 Thread Jun Rao
> Jun > > > > On Wed, Jan 21, 2015 at 12:38 PM, Raghu Udiyar > > wrote: > > > > > Hello, > > > > > > I have a 6 node kafka cluster (0.8.0) where partition reassignment > > doesn't > > > seem to work on a few partiti

Re: Partition reassignment doesn't work on few partitions

2015-01-21 Thread Raghu Udiyar
fixed some reassignment issues. > > Thanks, > > Jun > > On Wed, Jan 21, 2015 at 12:38 PM, Raghu Udiyar > wrote: > > > Hello, > > > > I have a 6 node kafka cluster (0.8.0) where partition reassignment > doesn't > > seem to work on a few partitions.

Re: Partition reassignment doesn't work on few partitions

2015-01-21 Thread Jun Rao
Any error in the controller and state-change log? Also, you may want to upgrade to 0.8.1, which fixed some reassignment issues. Thanks, Jun On Wed, Jan 21, 2015 at 12:38 PM, Raghu Udiyar wrote: > Hello, > > I have a 6 node kafka cluster (0.8.0) where partition reassignment doesn&#x

Partition reassignment doesn't work on few partitions

2015-01-21 Thread Raghu Udiyar
Hello, I have a 6 node kafka cluster (0.8.0) where partition reassignment doesn't seem to work on a few partitions. This happens within the same, as well as across other topics. Following is the behavior observed : 1. For a successful reassignment, the kafka-reassign-partitions.sh re

Re: Kafka Topic/Partition Reassignment issue

2014-12-17 Thread Neha Narkhede
The reassignment tool outputs the original assignment before executing the next one. If you have that saved, you can initiate another assignment to make it go to its initial state. That is probably a safer way to fix the reassignment. On Wed, Dec 17, 2014 at 3:12 PM, Salman Ahmed wrote: > > I had

Kafka Topic/Partition Reassignment issue

2014-12-17 Thread Salman Ahmed
I had an issue where one kafka node was filling up on disk space. I used the reassignment script in an incorrect way, overloading large number of topics/partition on two target machines, which caused kafka to stop on those machines. I would like to cancel the reassignment process, and restore it t

Re: Partition reassignment reversed

2014-12-08 Thread Jun Rao
drew Jorgensen > wrote: > >> I am using kafka 0.8. >> Yes I did run —verify, but got some weird output from it I had never >> seen before that looked something like: >> >> Status of partition reassignment: >> ERROR: Assigned replicas (5,2) don't

Re: Partition reassignment reversed

2014-12-04 Thread Andrew Jorgensen
from it I had never seen before that looked something like: Status of partition reassignment: ERROR: Assigned replicas (5,2) don't match the list of replicas for reassignment (5) for partition [topic-1,248] ERROR: Assigned replicas (7,3) don't match the list of replicas for reassignme

Re: Failed partition reassignment

2014-12-04 Thread Jun Rao
> > On Tue, Dec 2, 2014 at 11:20 PM, Jun Rao wrote: > > > Is there an easy way to reproduce the issues that you saw? > > > > Thanks, > > > > Jun > > > > On Mon, Dec 1, 2014 at 6:31 AM, Karol Nowak wrote: > > > > > Hi, > > >

Re: Partition reassignment reversed

2014-12-04 Thread Jun Rao
verify, but got some weird output from it I had never seen > before that looked something like: > > Status of partition reassignment: > ERROR: Assigned replicas (5,2) don't match the list of replicas for > reassignment (5) for partition [topic-1,248] > ERROR: Assigned replicas

Re: Failed partition reassignment

2014-12-02 Thread Karol Nowak
2, 2014 at 11:20 PM, Jun Rao wrote: > Is there an easy way to reproduce the issues that you saw? > > Thanks, > > Jun > > On Mon, Dec 1, 2014 at 6:31 AM, Karol Nowak wrote: > > > Hi, > > > > I observed some error messages / exceptions while running partiti

Re: Partition reassignment reversed

2014-12-02 Thread Andrew Jorgensen
I am using kafka 0.8. Yes I did run —verify, but got some weird output from it I had never seen before that looked something like: Status of partition reassignment: ERROR: Assigned replicas (5,2) don't match the list of replicas for reassignment (5) for partition [topic-1,248] ERROR: Ass

Re: Partition reassignment reversed

2014-12-02 Thread Jun Rao
in a 12 node cluster. A few weeks prior I > did a partition reassignment to add four new kafka brokers to the cluster. > This cluster has 4 topics on it each with 350 partitions each, a retention > policy of 6 hours, and a replication factor of 1. Originally I attempted to > run

Re: Failed partition reassignment

2014-12-02 Thread Jun Rao
Is there an easy way to reproduce the issues that you saw? Thanks, Jun On Mon, Dec 1, 2014 at 6:31 AM, Karol Nowak wrote: > Hi, > > I observed some error messages / exceptions while running partition > reassignment on kafka 0.8.1.1 cluster. Being fairly new to this system I

Partition reassignment reversed

2014-12-01 Thread Andrew Jorgensen
weeks prior I did a partition reassignment to add four new kafka brokers to the cluster. This cluster has 4 topics on it each with 350 partitions each, a retention policy of 6 hours, and a replication factor of 1. Originally I attempted to run a migration on all of the topics and partitions

Failed partition reassignment

2014-12-01 Thread Karol Nowak
Hi, I observed some error messages / exceptions while running partition reassignment on kafka 0.8.1.1 cluster. Being fairly new to this system I'm not sure if these indicate serious failures or transient problems, or if manual intervention is needed. I used kafka-reassign-partitions.

Re: Questions re: partition reassignment

2014-09-24 Thread Neha Narkhede
Okay, so just to clarify, if I have a partition where the leader is broker 0, the ISR is [0, 1] and I make a partition reassignment with a new AR list of [1, 0], broker 1 won't take over leadership? I was under the impression that the "preferred" replica would become the leader

Re: Questions re: partition reassignment

2014-09-22 Thread Mathias Söderberg
> > Partition reassignment will not move the leader unless the old leader is > not part of the new set of replicas. > Even when it does move the leader, it waits until the new replicas enter > the ISR. Okay, so just to clarify, if I have a partition where the leader is broker 0, t

Re: Questions re: partition reassignment

2014-09-22 Thread Neha Narkhede
My current interpretation is that if I start a partition reassignment, for the sake of simplicity let's assume it's just for a single partition, the new leader will first become a follower of the current leader, and when it has caught up it'll transfer leadership over to its

Questions re: partition reassignment

2014-09-21 Thread Mathias Söderberg
ving partitions between brokers. My current interpretation is that if I start a partition reassignment, for the sake of simplicity let's assume it's just for a single partition, the new leader will first become a follower of the current leader, and when it has caught up it'll transfer le

Kafka 0.8.1.1 partition reassignment script

2014-06-25 Thread Florian Dambrine
Hi, I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like to add a new broker in the cluster. Each kafka instance has 400Go of data and we are using replication factor equals to 3 with 50 partitions for each topics. I have check the documentation and especially the section d

Kafka partition reassignment script

2014-06-25 Thread Florian Dambrine
Hi, I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like to add a new broker in the cluster. Each kafka instance has 400Go of data and we are using replication factor equals to 3 with 50 partitions for each topics. I have check the documentation and especially the section d

Fwd: Kafka 0.8.1.1 partition reassignment script

2014-06-25 Thread Florian Dambrine
Hi, I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like to add a new broker in the cluster. Each kafka instance has 400Go of data and we are using replication factor equals to 3 with 50 partitions for each topics. I have check the documentation and especially the section d

Re: restarting a broker during partition reassignment

2014-06-25 Thread Guozhang Wang
< luke.foreh...@networkedinsights.com> wrote: > > My hypothesis for how Partition [luke3,3] with leader 11, had offset reset > to zero, caused by reboot of leader broker during partition reassignment: > > The replicas for [luke3,3] were in progress being reassigned from broker >

Re: restarting a broker during partition reassignment

2014-06-24 Thread Luke Forehand
My hypothesis for how Partition [luke3,3] with leader 11, had offset reset to zero, caused by reboot of leader broker during partition reassignment: The replicas for [luke3,3] were in progress being reassigned from broker 10,11,12 -> 11,12,13 I rebooted broker 11 which was the leader

Re: restarting a broker during partition reassignment

2014-06-23 Thread Guozhang Wang
-beta. I have a 4 node cluster with one broker per node, and a > topic with 8 partitions and 3 replicas. Each partition has about 6 > million records. > > I generated a partition reassignment json that basically causes all > partitions to be shifted by one broker. As the reassig

restarting a broker during partition reassignment

2014-06-23 Thread Luke Forehand
Hello, I am testing kafka 0.8.1.1 in preparation for an upgrade from kafka-0.8.1-beta. I have a 4 node cluster with one broker per node, and a topic with 8 partitions and 3 replicas. Each partition has about 6 million records. I generated a partition reassignment json that basically causes all

Re: Too many replicas after partition reassignment

2014-04-21 Thread Jun Rao
Which version of Kafka are you using? Thanks, Jun On Mon, Apr 21, 2014 at 11:41 AM, Ryan Berdeen wrote: > After doing some partition reassignments, I've ended up with some > partitions that have both the old and new brokers assigned. > > The output of kafka-topics.sh --describe looks like thi

Re: Too many replicas after partition reassignment

2014-04-21 Thread Timothy Chen
Hi Ryan, Also KAFKA-1317 should be fixed in both trunk and latest 0.8.1 branch, are you running with either or just with one of the previous released versions? Tim On Mon, Apr 21, 2014 at 5:00 PM, Guozhang Wang wrote: > Hi Ryan, > > Did you see any error logs on the new controller's controller

Re: Too many replicas after partition reassignment

2014-04-21 Thread Guozhang Wang
Hi Ryan, Did you see any error logs on the new controller's controller log and state-change log? Guozhang On Mon, Apr 21, 2014 at 11:41 AM, Ryan Berdeen wrote: > After doing some partition reassignments, I've ended up with some > partitions that have both the old and new brokers assigned. > >

Too many replicas after partition reassignment

2014-04-21 Thread Ryan Berdeen
After doing some partition reassignments, I've ended up with some partitions that have both the old and new brokers assigned. The output of kafka-topics.sh --describe looks like this: Topic:cs-es-indexer-a PartitionCount:30 ReplicationFactor:2 Configs: retention.ms=1080 ... Topic: cs-es-ind

Re: partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
On Thu, Dec 12, 2013 at 9:46 PM, Jun Rao wrote: > Since we don't support delete topics yet, you would have to wipe out all ZK > and kafka logs. > > Thanks, > > Jun > > got it and done. so it sounds like i should run a number of disparate clusters to spread risk for topics since a partition is an

Re: partition reassignment stuck on 0.8

2013-12-12 Thread Jun Rao
Since we don't support delete topics yet, you would have to wipe out all ZK and kafka logs. Thanks, Jun On Thu, Dec 12, 2013 at 9:32 PM, David Birdsong wrote: > On Thu, Dec 12, 2013 at 9:28 PM, Jun Rao wrote: > > > Could you try starting from scratch again? The recent fix that we had may > >

Re: partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
On Thu, Dec 12, 2013 at 9:28 PM, Jun Rao wrote: > Could you try starting from scratch again? The recent fix that we had may > not be able to recover a cluster already in an inconsistent state. > > Thanks, > > Jun > > > How does one start from scratch? Wipe ZK, is there some state file? I have oth

Re: partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
On Thu, Dec 12, 2013 at 9:28 PM, Guozhang Wang wrote: > David, > > Could you try to see if this is due to > https://issues.apache.org/jira/browse/KAFKA-1178? > > Guozhang > Which node do I look for this on? Leader? ISR-candidate? Controller? > > > On Thu, Dec 12, 2013 at 8:45 PM, David Birdson

Re: partition reassignment stuck on 0.8

2013-12-12 Thread Guozhang Wang
David, Could you try to see if this is due to https://issues.apache.org/jira/browse/KAFKA-1178? Guozhang On Thu, Dec 12, 2013 at 8:45 PM, David Birdsong wrote: > I was running a 2-node kafka cluster off github trunnk at: > eedbea6526986783257ad0e025c451a8ee3d9095 > > ...for a few weeks with no

Re: partition reassignment stuck on 0.8

2013-12-12 Thread Jun Rao
Could you try starting from scratch again? The recent fix that we had may not be able to recover a cluster already in an inconsistent state. Thanks, Jun On Thu, Dec 12, 2013 at 8:45 PM, David Birdsong wrote: > I was running a 2-node kafka cluster off github trunnk at: > eedbea6526986783257ad0e

partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
I was running a 2-node kafka cluster off github trunnk at: eedbea6526986783257ad0e025c451a8ee3d9095 ...for a few weeks with no issues. I recently downloaded the 0.8 stable version, configured/started two new brokers with 0.8. I successfully reassigned all but 1 partition from the older pair to th

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
al release? Any error in controller > log? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Fri, Dec 6, 2013 at 4:38 PM, Maxime N

Re: Partition reassignment stucked

2013-12-09 Thread Neha Narkhede
; > > > Jun > > > > > > > > > > > > > > > On Fri, Dec 6, 2013 at 4:38 PM, Maxime Nay > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > &g

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
gt; > > > > > > > Hi, > > > > > > > > > > We are trying to add a broker to a 10 node cluster. We have 7 > > different > > > > > topics, each of them is divided in 10 partitions, and their > > replication > > > >

Re: Partition reassignment stucked

2013-12-09 Thread Neha Narkhede
ave 7 > different > > > > topics, each of them is divided in 10 partitions, and their > replication > > > > factor is 3. > > > > > > > > To send traffic to this new node, we tried the > > > kafka-reassign-partitions.sh > > > >

  1   2   >