Replying to myself I ended by making a small python utility that do the
work.
On 12/08/2022 12:50, Raphael Mazelier wrote:
Hi kafka community,
I've been an happy user of kafka since years. Today I faced a problem
that begins to make mad...
So we have the project to migrate from and old (and unmonitored,
etc..) kafka cluster to multiple new fresh clusters. Great.
So the plan was clear, identify source topics and source groups, use
mirromarker2 with whitelist configuration and migrate consumer by
consumer on the fresh cluster, then producer.
For that we need that consumer group should be synced at some point
(if not every consumer will restart from its own strategy).
That were the problem begins. Mirrormaker2 seems to works well
replicating topics, but on consumer it's very erratic to say the
least. My temporary conclusion is that mm2 only "replicate" empty
consumer group (which can make sense?), but inspecting the offset show
me that they are not sync.
I tried to dig into config/log and I never find something that works
reliably. (btw the documentation of mirrormaker2 is just horrible, we
don't what config stanza is supposed to be applied, and reading the
log is .. hmm ... ).
I also try confluent replicator; which is also replicating well
topics, but not consumer group.
So first question:
- is someone have already succeed making work mm2 with consumer group
translation/replication?
B Plan:
Assuming I can shut my consumers for some time, assuming also I have
the exact same topics/part topology so I don't translation.
I have in the idea to find a way to take manually a snapshot of each
consumer group involved, and recreate them on the destination cluster.
It should be possible (after all mm2 is supposed to do it); and after
all it's just a matter of manually committing offset by partition.
kafka-topics propose an option to reset offset to earliest or a time
(maybe its sufficient we will see) ; but I would have prefer a tool to
create the exact same consumer group in the dest cluster.
Is someone have already done that? and know an tool to do it?
Any help appreciated :)
--
Raphael Mazelier
