Done, and things look clean in Zookeeper now. I'm guessing that it is safe for me to try to increase the replication count on this topic again? :)
Any idea what might have happened to put me in this state? I had successfully increased probably 10 or so topics, but then I started getting this message and nothing worked after that. Again, thank you so much for your time and knowledge. Very much appreciated. Chris On Sun, May 17, 2015 at 2:20 PM, Clark Haskins <cl...@kafka.guru> wrote: > No problem. > > Delete the reassign_partitions znode. Then do the controller again. > > The controller znode gets instantly re-created but with a different broker > as the controller. > > If you do a get /controller you can see the values change > > -Clark > > Sent from my iPhone > > > On May 17, 2015, at 11:48 AM, Chris Neal <cwn...@gmail.com> wrote: > > > > Gah. Sorry for the spamming replies. > > > > The reassign_partitions znode is also still there, and still empty....For > > what that's worth. :) > > Back to thinking that deleting the reassign_partitions znode is the right > > way to go. > > > > Chris > > > >> On Sun, May 17, 2015 at 1:46 PM, Chris Neal <cwn...@gmail.com> wrote: > >> > >> Ok, I might have lied. Timezone got me. I do see this in the Kafka > logs, > >> so maybe it did work? > >> > >> [2015-05-17 14:36:57,924] INFO 0 successfully elected as leader > >> (kafka.server.ZookeeperLeaderElector) > >> [2015-05-17 14:36:57,924] INFO 0 successfully elected as leader > >> (kafka.server.ZookeeperLeaderElector) > >> [2015-05-17 14:36:58,975] INFO New leader is 0 > >> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > >> [2015-05-17 14:36:58,975] INFO New leader is 0 > >> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > >> > >> Chris > >> > >>> On Sun, May 17, 2015 at 1:43 PM, Chris Neal <cwn...@gmail.com> wrote: > >>> > >>> Thank you again. :) > >>> > >>> I ran the delete, but I'm not sure if it did anything: > >>> > >>> root> > >>> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>> -server myhost.mydomain.com:2181 delete /controller > >>> Connecting to myhost.mydomain.com:2181 > >>> > >>> WATCHER:: > >>> > >>> WatchedEvent state:SyncConnected type:None path:null > >>> root> > >>> > >>> I followed that up immediately by an "ls /", and the controller znode > was > >>> there. Either still there, or had already been re-created. There was > >>> nothing in the Kafka logs indicating that anything had happened > either, so > >>> I'm thinking maybe I need to try and delete the znode partition > manually? > >>> > >>> Or, should I have seen the controller znode disappear for a time? > >>> > >>> Very much appreciate your help! > >>> Chris > >>> > >>>> On Sun, May 17, 2015 at 1:33 PM, Clark Haskins <cl...@kafka.guru> > wrote: > >>>> > >>>> Yeah it's empty. So I would try and force a new controller election to > >>>> see if that cleans up the znode. > >>>> > >>>> Delete /controller > >>>> > >>>> If that doesn't work you will have to delete the reassign_partitions > >>>> znode manually > >>>> > >>>> -Clark > >>>> > >>>> Sent from my iPhone > >>>> > >>>>> On May 17, 2015, at 11:16 AM, Chris Neal <cwn...@gmail.com> wrote: > >>>>> > >>>>> Sorry :) > >>>>> Here you go: > >>>>> > >>>>> root> > >>>> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>>>> -server myhost.mydomain.com <http://myhost.mydomain.com:2181/>:2181 > >>>> get > >>>>> /admin/reassign_partitions > >>>>> Connecting to myhost.mydomain.com <http://myhost.mydomain.com:2181/ > >>>>> :2181 > >>>>> <snip> > >>>>> WATCHER:: > >>>>> > >>>>> WatchedEvent state:SyncConnected type:None path:null > >>>>> {"version":1,"partitions":[]} > >>>>> cZxid = 0x6001afde9 > >>>>> ctime = Wed Jan 21 18:37:40 UTC 2015 > >>>>> mZxid = 0x6001afde9 > >>>>> mtime = Wed Jan 21 18:37:40 UTC 2015 > >>>>> pZxid = 0x6001afde9 > >>>>> cversion = 0 > >>>>> dataVersion = 0 > >>>>> aclVersion = 0 > >>>>> ephemeralOwner = 0x0 > >>>>> dataLength = 29 > >>>>> numChildren = 0 > >>>>> > >>>>> Thank you! > >>>>> Chris > >>>>> > >>>>>> On Sun, May 17, 2015 at 12:58 PM, Clark Haskins <cl...@kafka.guru> > >>>> wrote: > >>>>>> > >>>>>> Do a get /admin/reassign_partitions > >>>>>> > >>>>>> -Clark > >>>>>> > >>>>>> Sent from my iPhone > >>>>>> > >>>>>>> On May 17, 2015, at 10:20 AM, Chris Neal <cwn...@gmail.com> wrote: > >>>>>>> > >>>>>>> Sure thing :) > >>>>>>> Hopefully I did this right. Somewhat of a Zookeeper noob. > >>>>>>> > >>>>>>> root> > >>>> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>>>>>> -server myhost.mydomain.com <http://myhost.mydomain.com:2181/ > >:2181 > >>>> ls > >>>>>>> /admin/reassign_partitions > >>>>>>> Connecting to myhost.mydomain.com < > http://myhost.mydomain.com:2181/ > >>>>>>> :2181 > >>>>>>> <snip> > >>>>>>> WATCHER:: > >>>>>>> > >>>>>>> WatchedEvent state:SyncConnected type:None path:null > >>>>>>> [] > >>>>>>> > >>>>>>> Hope that is helpful :) > >>>>>>> If this is not what you were asking for, please just let me know. > >>>>>>> Thank you! > >>>>>>> Chris > >>>>>>> > >>>>>>>> On Sun, May 17, 2015 at 12:17 PM, Clark Haskins <cl...@kafka.guru > > > >>>>>> wrote: > >>>>>>>> > >>>>>>>> The reassign_partitions znode is the important one. Please paste > the > >>>>>>>> contents of it. That node should only exist while there is a > >>>>>> reassignment > >>>>>>>> in progress. > >>>>>>>> > >>>>>>>> You can probably fix this up by forcing a new controller to come > >>>> online > >>>>>> by > >>>>>>>> deleting /controller > >>>>>>>> > >>>>>>>> -Clark > >>>>>>>> > >>>>>>>> Sent from my iPhone > >>>>>>>> > >>>>>>>>> On May 17, 2015, at 10:14 AM, Chris Neal <cwn...@gmail.com> > wrote: > >>>>>>>>> > >>>>>>>>> Hi Clark, > >>>>>>>>> > >>>>>>>>> Thank you for your reply! I do see that znode under /admin: > >>>>>>>>> > >>>>>>>>> root> > >>>> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>>>>>>>> -server myhost.mydomain.com:2181 ls /admin > >>>>>>>>> Connecting to myhost.mydomain.com:2181 > >>>>>>>>> <snip> > >>>>>>>>> WATCHER:: > >>>>>>>>> > >>>>>>>>> WatchedEvent state:SyncConnected type:None path:null > >>>>>>>>> [reassign_partitions, delete_topics] > >>>>>>>>> > >>>>>>>>> I'm not sure what this tells me though :) > >>>>>>>>> Again, thanks for your time. > >>>>>>>>> Chris > >>>>>>>>> > >>>>>>>>>> On Sun, May 17, 2015 at 12:20 AM, Clark Haskins > <cl...@kafka.guru > >>>>> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Does the partition reassignment znode exist under /admin in > >>>> zookeeper? > >>>>>>>>>> > >>>>>>>>>> -Clark > >>>>>>>>>> > >>>>>>>>>> Sent from my iPhone > >>>>>>>>>> > >>>>>>>>>>> On May 16, 2015, at 7:16 PM, Chris Neal <cwn...@gmail.com> > >>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Sorry for bumping my own thread. :S Just wanted to get it in > >>>> front > >>>>>> of > >>>>>>>>>> some > >>>>>>>>>>> eyes again! > >>>>>>>>>>> > >>>>>>>>>>> Thanks for your time and help. > >>>>>>>>>>> Chris > >>>>>>>>>>> > >>>>>>>>>>>> On Thu, May 14, 2015 at 10:24 AM, Chris Neal < > cwn...@gmail.com> > >>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi All, > >>>>>>>>>>>> > >>>>>>>>>>>> I am running: kafka_2.10-0.8.1.1, and when I run the > >>>>>>>>>>>> reassign-partitions.sh script, I get this: > >>>>>>>>>>>> > >>>>>>>>>>>> Partitions reassignment failed due to Partition reassignment > >>>>>> currently > >>>>>>>>>> in > >>>>>>>>>>>> progress for Map(). Aborting operation > >>>>>>>>>>>> kafka.common.AdminCommandFailedException: Partition > reassignment > >>>>>>>>>> currently > >>>>>>>>>>>> in progress for Map(). Aborting operation > >>>>>>>>>>>> at > >>>> > kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:204) > >>>>>>>>>>>> at > >>>> > kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:124) > >>>>>>>>>>>> at > >>>> > kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:49) > >>>>>>>>>>>> at > >>>> > kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) > >>>>>>>>>>>> > >>>>>>>>>>>> I'm running it as: > >>>>>>>>>>>> /opt/kafka/kafka/bin/kafka-reassign-partitions.sh --zookeeper > >>>>>>>> $ZOOKEEPER > >>>>>>>>>>>> --reassignment-json-file ./$1.json --execute > >>>>>>>>>>>> > >>>>>>>>>>>> and my json file looks like: > >>>>>>>>>>>> {"version":1, > >>>>>>>>>>>> "partitions":[ > >>>>>>>>>>>> {"topic":"perf_mytopic_raw", "partition":0,"replicas":[0,1]}, > >>>>>>>>>>>> {"topic":"perf_mytopic_raw","partition":1,"replicas":[0,1]}]} > >>>>>>>>>>>> > >>>>>>>>>>>> The empty Map() looks suspicious, and I found another post > from > >>>>>> about > >>>>>>>> a > >>>>>>>>>>>> year ago where it was thought that this might be a bug? I > >>>> looked > >>>>>>>>>> through > >>>>>>>>>>>> JIRA and didn't see anything that looked like this issue was > >>>> created > >>>>>>>> or > >>>>>>>>>>>> fixed, so I wanted to ask again about it. > >>>>>>>>>>>> > >>>>>>>>>>>> Any ideas why this might be failing? There is definitely not > >>>>>> another > >>>>>>>>>>>> reassignment in progress. :) > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks very much for your time. > >>>>>>>>>>>> Chris > >> >
