Gah. Sorry for the spamming replies. The reassign_partitions znode is also still there, and still empty....For what that's worth. :) Back to thinking that deleting the reassign_partitions znode is the right way to go.
Chris On Sun, May 17, 2015 at 1:46 PM, Chris Neal <cwn...@gmail.com> wrote: > Ok, I might have lied. Timezone got me. I do see this in the Kafka logs, > so maybe it did work? > > [2015-05-17 14:36:57,924] INFO 0 successfully elected as leader > (kafka.server.ZookeeperLeaderElector) > [2015-05-17 14:36:57,924] INFO 0 successfully elected as leader > (kafka.server.ZookeeperLeaderElector) > [2015-05-17 14:36:58,975] INFO New leader is 0 > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > [2015-05-17 14:36:58,975] INFO New leader is 0 > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > > Chris > > On Sun, May 17, 2015 at 1:43 PM, Chris Neal <cwn...@gmail.com> wrote: > >> Thank you again. :) >> >> I ran the delete, but I'm not sure if it did anything: >> >> root> >> ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh >> -server myhost.mydomain.com:2181 delete /controller >> Connecting to myhost.mydomain.com:2181 >> >> WATCHER:: >> >> WatchedEvent state:SyncConnected type:None path:null >> root> >> >> I followed that up immediately by an "ls /", and the controller znode was >> there. Either still there, or had already been re-created. There was >> nothing in the Kafka logs indicating that anything had happened either, so >> I'm thinking maybe I need to try and delete the znode partition manually? >> >> Or, should I have seen the controller znode disappear for a time? >> >> Very much appreciate your help! >> Chris >> >> On Sun, May 17, 2015 at 1:33 PM, Clark Haskins <cl...@kafka.guru> wrote: >> >>> Yeah it's empty. So I would try and force a new controller election to >>> see if that cleans up the znode. >>> >>> Delete /controller >>> >>> If that doesn't work you will have to delete the reassign_partitions >>> znode manually >>> >>> -Clark >>> >>> Sent from my iPhone >>> >>> > On May 17, 2015, at 11:16 AM, Chris Neal <cwn...@gmail.com> wrote: >>> > >>> > Sorry :) >>> > Here you go: >>> > >>> > root> >>> > >>> ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh >>> > -server myhost.mydomain.com <http://myhost.mydomain.com:2181/>:2181 >>> get >>> > /admin/reassign_partitions >>> > Connecting to myhost.mydomain.com <http://myhost.mydomain.com:2181/ >>> >:2181 >>> > <snip> >>> > WATCHER:: >>> > >>> > WatchedEvent state:SyncConnected type:None path:null >>> > {"version":1,"partitions":[]} >>> > cZxid = 0x6001afde9 >>> > ctime = Wed Jan 21 18:37:40 UTC 2015 >>> > mZxid = 0x6001afde9 >>> > mtime = Wed Jan 21 18:37:40 UTC 2015 >>> > pZxid = 0x6001afde9 >>> > cversion = 0 >>> > dataVersion = 0 >>> > aclVersion = 0 >>> > ephemeralOwner = 0x0 >>> > dataLength = 29 >>> > numChildren = 0 >>> > >>> > Thank you! >>> > Chris >>> > >>> >> On Sun, May 17, 2015 at 12:58 PM, Clark Haskins <cl...@kafka.guru> >>> wrote: >>> >> >>> >> Do a get /admin/reassign_partitions >>> >> >>> >> -Clark >>> >> >>> >> Sent from my iPhone >>> >> >>> >>> On May 17, 2015, at 10:20 AM, Chris Neal <cwn...@gmail.com> wrote: >>> >>> >>> >>> Sure thing :) >>> >>> Hopefully I did this right. Somewhat of a Zookeeper noob. >>> >>> >>> >>> root> >>> >> >>> ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh >>> >>> -server myhost.mydomain.com <http://myhost.mydomain.com:2181/>:2181 >>> ls >>> >>> /admin/reassign_partitions >>> >>> Connecting to myhost.mydomain.com <http://myhost.mydomain.com:2181/ >>> >>> :2181 >>> >>> <snip> >>> >>> WATCHER:: >>> >>> >>> >>> WatchedEvent state:SyncConnected type:None path:null >>> >>> [] >>> >>> >>> >>> Hope that is helpful :) >>> >>> If this is not what you were asking for, please just let me know. >>> >>> Thank you! >>> >>> Chris >>> >>> >>> >>>> On Sun, May 17, 2015 at 12:17 PM, Clark Haskins <cl...@kafka.guru> >>> >> wrote: >>> >>>> >>> >>>> The reassign_partitions znode is the important one. Please paste the >>> >>>> contents of it. That node should only exist while there is a >>> >> reassignment >>> >>>> in progress. >>> >>>> >>> >>>> You can probably fix this up by forcing a new controller to come >>> online >>> >> by >>> >>>> deleting /controller >>> >>>> >>> >>>> -Clark >>> >>>> >>> >>>> Sent from my iPhone >>> >>>> >>> >>>>> On May 17, 2015, at 10:14 AM, Chris Neal <cwn...@gmail.com> wrote: >>> >>>>> >>> >>>>> Hi Clark, >>> >>>>> >>> >>>>> Thank you for your reply! I do see that znode under /admin: >>> >>>>> >>> >>>>> root> >>> >> >>> ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh >>> >>>>> -server myhost.mydomain.com:2181 ls /admin >>> >>>>> Connecting to myhost.mydomain.com:2181 >>> >>>>> <snip> >>> >>>>> WATCHER:: >>> >>>>> >>> >>>>> WatchedEvent state:SyncConnected type:None path:null >>> >>>>> [reassign_partitions, delete_topics] >>> >>>>> >>> >>>>> I'm not sure what this tells me though :) >>> >>>>> Again, thanks for your time. >>> >>>>> Chris >>> >>>>> >>> >>>>>> On Sun, May 17, 2015 at 12:20 AM, Clark Haskins <cl...@kafka.guru >>> > >>> >>>> wrote: >>> >>>>>> >>> >>>>>> Does the partition reassignment znode exist under /admin in >>> zookeeper? >>> >>>>>> >>> >>>>>> -Clark >>> >>>>>> >>> >>>>>> Sent from my iPhone >>> >>>>>> >>> >>>>>>> On May 16, 2015, at 7:16 PM, Chris Neal <cwn...@gmail.com> >>> wrote: >>> >>>>>>> >>> >>>>>>> Sorry for bumping my own thread. :S Just wanted to get it in >>> front >>> >> of >>> >>>>>> some >>> >>>>>>> eyes again! >>> >>>>>>> >>> >>>>>>> Thanks for your time and help. >>> >>>>>>> Chris >>> >>>>>>> >>> >>>>>>>> On Thu, May 14, 2015 at 10:24 AM, Chris Neal <cwn...@gmail.com> >>> >>>> wrote: >>> >>>>>>>> >>> >>>>>>>> Hi All, >>> >>>>>>>> >>> >>>>>>>> I am running: kafka_2.10-0.8.1.1, and when I run the >>> >>>>>>>> reassign-partitions.sh script, I get this: >>> >>>>>>>> >>> >>>>>>>> Partitions reassignment failed due to Partition reassignment >>> >> currently >>> >>>>>> in >>> >>>>>>>> progress for Map(). Aborting operation >>> >>>>>>>> kafka.common.AdminCommandFailedException: Partition reassignment >>> >>>>>> currently >>> >>>>>>>> in progress for Map(). Aborting operation >>> >>>>>>>> at >>> >> >>> kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:204) >>> >>>>>>>> at >>> >> >>> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:124) >>> >>>>>>>> at >>> >> >>> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:49) >>> >>>>>>>> at >>> >> >>> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) >>> >>>>>>>> >>> >>>>>>>> I'm running it as: >>> >>>>>>>> /opt/kafka/kafka/bin/kafka-reassign-partitions.sh --zookeeper >>> >>>> $ZOOKEEPER >>> >>>>>>>> --reassignment-json-file ./$1.json --execute >>> >>>>>>>> >>> >>>>>>>> and my json file looks like: >>> >>>>>>>> {"version":1, >>> >>>>>>>> "partitions":[ >>> >>>>>>>> {"topic":"perf_mytopic_raw", "partition":0,"replicas":[0,1]}, >>> >>>>>>>> {"topic":"perf_mytopic_raw","partition":1,"replicas":[0,1]}]} >>> >>>>>>>> >>> >>>>>>>> The empty Map() looks suspicious, and I found another post from >>> >> about >>> >>>> a >>> >>>>>>>> year ago where it was thought that this might be a bug? I >>> looked >>> >>>>>> through >>> >>>>>>>> JIRA and didn't see anything that looked like this issue was >>> created >>> >>>> or >>> >>>>>>>> fixed, so I wanted to ask again about it. >>> >>>>>>>> >>> >>>>>>>> Any ideas why this might be failing? There is definitely not >>> >> another >>> >>>>>>>> reassignment in progress. :) >>> >>>>>>>> >>> >>>>>>>> Thanks very much for your time. >>> >>>>>>>> Chris >>> >> >>> >> >> >
