Resolution to this issue was to restart the broker which was the active controller so that that role move to a new broker. Not clear why the existing active controller was no longer getting re-assignment updates from the zookeeper quorum.
On Wed, Jul 10, 2019 at 5:43 PM Devin Kramer <dkra...@rubiconproject.com> wrote: > We have a situation on one of our clusters where when we run a partition > reassignment using /usr/local/kafka/bin/kafka-reassign-partitions.sh we see > the reassignment data published to the zookeeper in > /admin/reassign_partitions and all partitions requests show in progress but > the brokers do not appear to be seeing the "request" and thus the requested > reassignments all remain stuck "in progress" and nothing is moving. This > is a process we have used before on this cluster and use often across our > clusters but this is the first time we have seen this. The cluster seems > healthy in all other respects. We have validated that the json input we are > passing to the reassign script is correct. > > OS CentOS 6.7 > Kafka v2.0.0-1 > Zookeeper v3.4.6-3 > > Any idea on what could cause this or ways to troubleshoot further would be > appreciated. > > Devin Kramer > > -- Devin Kramer Senior Director, Infrastructure Engineering 12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094 RUBICONPROJECT.COM <http://www.rubiconproject.com> | JOIN US <http://rubiconproject.com/join-us/> | FOLLOW US <http://twitter.com/rubiconproject>
