Not sure if you saw this before, but you might be interested in some of the
work Uber has done for inter-cluster replication and federation. They of
course use their own tool, uReplicator (https://github.com/uber/uReplicator),
instead of mirror maker, but you should be able to draw the same insight
You can either use the built-in kafka-reassign-partitions.sh script (
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-4.ReassignPartitionsTool
)
Or in industry others use tooling such as
https://github.com/Yelp/kafka-utils (more easy to use scripts) or automate
There's lots of open source tooling for this! Some examples, but there's
plenty more:
https://github.com/Yelp/kafka-utils (more easy to use scripts)
https://github.com/linkedin/cruise-control (automated system)
https://github.com/DataDog/kafka-kit
On Wed, Feb 19, 2020 at 4:52 PM Maurício Linhare
Hey Vincent, I think you need to set both the configs for the brokers and
the individual partitions you are moving themselves.
For an automated system that can make this easier, check out
https://github.com/DataDog/kafka-kit/tree/master/cmd/autothrottle, though
it requires DataDog, you can use a s
hich topic/partition is replicated.
>
> Is that correct ?
>
> On Thu, Feb 20, 2020, at 22:09, Brian Sang wrote:
> > Hey Vincent, I think you need to set both the configs for the brokers and
> > the individual partitions you are moving themselves.
> >
> > For an automa
to recap, it's enough to set follower.replication.throttled.replicas
> for every partition assigned to that broker ? I'm assuming that during
> bootstrap the broker is a follower for everything.
>
> On Fri, Feb 21, 2020, at 22:09, Brian Sang wrote:
> > You can get the list o
If the schema is no longer compatible I think it makes sense to create a
new topic.
Offsets wise you can just have the consumer finish consuming from the old
topic/schema (i.e. migrate all producers to use the new schema format
first) and then migrate consumers to use the new schema format (and th
