We had to revert to 0.8.3 because three of our topics seem to have gotten corrupted during the upgrade. As soon as we did the upgrade producers to the three topics I mentioned stopped being able to do writes. The clients complained (occasionally) about leader not found exceptions. We restarted our clients and brokers but that didn't seem to help. Actually even after reverting to 0.8.3 these three topics were broken. To fix it we had to stop all clients, delete the topics, create them again and then restart the clients.
I realize this is not a lot of info. I couldn't wait to get more debug info because the cluster was actually being used. Has any one run into something like this? Are there any known issues with old consumers/producers. The topics that got busted had clients writing to them using the old Java wrapper over the Scala producer. Here are the steps I took to upgrade. For each broker: 1. Stop the broker. 2. Restart with the 0.9 broker running with inter.broker.protocol.version=0.8.2.X 3. Wait for under replicated partitions to go down to 0. 4. Go to step 1. Once all the brokers were running the 0.9 code with inter.broker.protocol.version=0.8.2.X we restarted them one by one with inter.broker.protocol.version=0.9.0.0 When reverting I did the following. For each broker. 1. Stop the broker. 2. Restart with the 0.9 broker running with inter.broker.protocol.version=0.8.2.X 3. Wait for under replicated partitions to go down to 0. 4. Go to step 1. Once all the brokers were running 0.9 code with inter.broker.protocol.version=0.8.2.X I restarted them one by one with the 0.8.2.3 broker code. This however like I mentioned did not fix the three broken topics. On Mon, Dec 14, 2015 at 3:13 PM, Rajiv Kurian <ra...@signalfx.com> wrote: > Now that it has been a bit longer, the spikes I was seeing are gone but > the CPU and network in/out on the three brokers that were showing the > spikes are still much higher than before the upgrade. Their CPUs have > increased from around 1-2% to 12-20%. The network in on the same brokers > has gone up from under 2 Mb/sec to 19-33 Mb/sec. The network out has gone > up from under 2 Mb/sec to 29-42 Mb/sec. I don't see a corresponding > increase in kafka messages in per second or kafka bytes in per second JMX > metrics. > > Thanks, > Rajiv >
