Re: mm2: size of target vs source topic

2021-10-04 Thread Urbán Dániel
My guess is that MM2 decompressed, batched, then compressed the data again in bigger chunks - in case the batches in the source were small, MM2 could be able to improve the compression with bigger batches. 2021.10.04. 19:21 keltezéssel, Men Lim írta: bump On Wed, Sep 29, 2021 at 4:09 PM Men L

Re: MirrorMaker 2 without write access on source

2022-01-18 Thread Urbán Dániel
Hi Ruben, The offset-syncs topic is necessary when the checkpointing feature of MM2 is being used (a.k.a. consumer group offset replication). Since the offsets of the original and the replica topic are not guaranteed to exactly match, there has to be a translation step when replicating the com

Re: Connect task configuration limits?

2022-05-21 Thread Urbán Dániel
Hi Ryan, There are some limits, as the configs are stored inside an internal topic of Connect. So the usual message size and producer request size limitations apply. You can reconfigure the internal topic to allow larger messages than the default (I think it's 1MB), and the producer max reques

Re: MM2: Failed to reconfigure connector's tasks (MirrorCheckpointConnector), retrying after backoff

2022-08-20 Thread Urbán Dániel
Hi, Are you using multiple MM2 instances? If yes, there is a known issue around that, the Connect REST API is missing from MM2 instances, causing clusters to fail. There is a dormant KIP trying to fix it: https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distribut