Hi,
Thanks for your help. It is part of our plan to move forward to 0.10,
but we need to do it one step at a time. And rebalancing topics,
including __consumer_offsets, are on the top as we had incidents
associated with it.
-- Anderson
On 14/07/2016 13:59, Tom Crayford wrote:
Also note that there were a number of bugs fixed in the log cleaner thread
between 0.8 and the latest release. I wouldn't be comfortable relying on
kafka committed offsets on a version under 0.10 for a new production
system, and would carefully consider an upgrade all the way to the latest
release.
On Thu, Jul 14, 2016 at 1:37 PM, Todd Palino <tpal...@gmail.com> wrote:
It's safe to move the partitions of the offsets topic around. You'll move
the consumer coordinators as you do, however, so the one thing you want to
make sure of, especially running an older version, is that log compaction
is working on your brokers and those partitions have been compacted. The
coordinator needs to bootstrap the topic, and if log compaction is broken
that can take a very long time. During that time, it will return errors to
consumers for offset operations, and that can cause offset resets.
-Todd
On Thursday, July 14, 2016, Anderson Goulart <anderson.goul...@boxever.com
wrote:
Hi,
I am running kafka 0.8.2.1 under aws instances with multiple availability
zones. As we want a rack aware partition replication, we have our own
partition layout distribution, to make sure all partitions are well
balanced between nodes, leaders and availability zones.
The problem arises with __consumer_offsets internal topic. In our current
environment it has 50 partitions and are all under the same AZ, with an
unbalanced leader (all wrong!)
The question is: should I manually change its partition layout
distribution as I do for the other topics? Is it safe to reassign the new
layout for this internal topic, using kafka-reassign-partitions.sh?
Thanks, Anderson
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
