Re: Mirror Maker bidirectional offset sync

Hey Jeroen, Thanks for looking into the collision theory! > There are no offset collisions as the offset-syncs albeit being stored on > the same cluster, offsets from A->B are stored > in mm2-offset-syncs.b.internal whereas offsets from B->A are stored > in mm2-offset-syncs.a.internal. The offse

Re: Mirror Maker bidirectional offset sync

Ryanne, > > b.a.replicate-me-0 > That's actually impossible with MM2. Thanks, I see the isCycle check in MirrorSourceConnector. That makes me even more curious how the renameTopicPartition method triggers without a change such as the one that Jeroen has prototyped, since the only thing that emits

Re: Mirror Maker bidirectional offset sync

Hey Greg, There are no offset collisions as the offset-syncs albeit being stored on the same cluster, offsets from A->B are stored in mm2-offset-syncs.b.internal whereas offsets from B->A are stored in mm2-offset-syncs.a.internal. What's curious though is the B->A checkpoint connector (which has of

Re: Mirror Maker bidirectional offset sync

> b.a.replicate-me-0 That's actually impossible with MM2. It won't allow such topics. It would be pointless to replicate data from A back to A, or from B back to B. The data is already there. MM2's replication logic is significantly more advanced than just adding a prefix. Tho it does appear that

Re: Mirror Maker bidirectional offset sync

Hey Jeroen, Thanks for sharing your prototype! It is very interesting! > I couldn't reproduce your hypothesis. I think my hypothesis was for another setup which didn't involve code changes, and instead relied on A->B->A round trip replication to produce the "backwards" offset syncs. I believe th

Re: Mirror Maker bidirectional offset sync

I see, makes complete sense to me. I built a custom version [1] based off of Kafka 3.5.1 with bidirectional offset replication enabled so I could do some more testing. Offset translation back upstream works well; I think because of the reason Ryanne pointed out, both topics contain identical data.

Re: Mirror Maker bidirectional offset sync

Hey Ryanne, Thanks for the context, but I still don't see the situation where this function is helpful. Also "A's topic1 and B's a.topic1 should be the same data (minus replication lag)." isn't true in the presence of failures/hard restarts, compaction, and transaction markers. Thanks, Greg On

Re: Mirror Maker bidirectional offset sync

> do you recall the purpose of [...] renameTopicPartition [?] A's topic1 and B's a.topic1 should be the same data (minus replication lag). You can't consume a record in a.topic1 that hasn't been replicated yet -- a remote topic by definition does not have any records that MM2 didn't put there. So

Re: Mirror Maker bidirectional offset sync

Hi Jeroen, I'm glad you're experimenting with MM2, and I hope we can give you some more context to explain what you're seeing. > I wrote a small program to produce these offset syncs for the prefixed > topic, and this successfully triggers the Checkpoint connector to start > replicating the consu

Re: Mirror Maker bidirectional offset sync

Thank you both for your swift responses! Ryanne, the MirrorConnectorsIntegrationBaseTest only tests offset replication in cases where the producer migrated to the secondary cluster as well, starts feeding messages into the non-prefixed topic which are subsequently consumed by the consumer on the s

Re: Mirror Maker bidirectional offset sync

Jeroen, MirrorClient will correctly translate offsets for both failover and failback, exactly as you describe. It's possible to automate failover and failback using that logic. The integration tests automatically fail over and fail back, for example. I've seen it done two ways: during startup withi

Re: Mirror Maker bidirectional offset sync

Hi Jeroen, Thanks for looking into MM2 for your use-case! I believe the "active-active" model that MM2 uses is really more of a pair of opposing "active-passive" replication flows, and this extends to the offset sync mechanism too. MM2 doesn't change the state of the source topic or consumer grou

Re: Mirror Maker 2 - offset sync from source to target

ocumentation explaining the same ? > I tried to find a lot in the past > > > From: Greg Harris > Sent: Monday, October 23, 2023 11:23:26 PM > To: users@kafka.apache.org > Subject: Re: Mirror Maker 2 - offset sync from source to tar

Re: Mirror Maker 2 - offset sync from source to target

Not a problem Greg. Is there some documentation explaining the same ? I tried to find a lot in the past From: Greg Harris Sent: Monday, October 23, 2023 11:23:26 PM To: users@kafka.apache.org Subject: Re: Mirror Maker 2 - offset sync from source to target [You

Re: Mirror Maker 2 - offset sync from source to target

larification plz > > > > > > In bullet for You mention "4. The target group does not exist, or has > no active consumers" > > > If group on target does not exist, will it be created without active > consumers ? > > > > > > -Original Mess

Re: Mirror Maker 2 - offset sync from source to target

es not exist, will it be created without active > > consumers ? > > > > -Original Message- > > From: Greg Harris > > Sent: Monday, October 23, 2023 8:56 PM > > To: users@kafka.apache.org > > Subject: Re: Mirror Maker 2 - offset sync from s

Re: Mirror Maker 2 - offset sync from source to target

n "4. The target group does not exist, or has no > active consumers" > If group on target does not exist, will it be created without active > consumers ? > > -Original Message- > From: Greg Harris > Sent: Monday, October 23, 2023 8:56 PM > To: users@kafk

RE: Mirror Maker 2 - offset sync from source to target

8:56 PM To: users@kafka.apache.org Subject: Re: Mirror Maker 2 - offset sync from source to target [You don't often get email from greg.har...@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] CAUTION: This email is from an external source. Please

Re: Mirror Maker 2 - offset sync from source to target

--Original Message- > From: Greg Harris > Sent: Monday, October 23, 2023 8:42 PM > To: users@kafka.apache.org > Subject: Re: Mirror Maker 2 - offset sync from source to target > > CAUTION: This email is from an external source. Please don’t open any unknown > links or attach

RE: Mirror Maker 2 - offset sync from source to target

Hi Greg, Thank you very much, it is the most detailed answer I would expect. -Original Message- From: Greg Harris Sent: Monday, October 23, 2023 8:42 PM To: users@kafka.apache.org Subject: Re: Mirror Maker 2 - offset sync from source to target CAUTION: This email is from an external

Re: Mirror Maker 2 - offset sync from source to target

Hi Alexander, Thanks for using MirrorMaker2! If you turn on `sync.group.offsets.enabled`, then the MirrorCheckpointTask will sync the offsets if all of the following is true: 1. The source group exists 2. The source group name matches the configured group filter (group.filter.class, groups, group

Re: mirror-maker

hi Richard, Am 02.10.23 um 16:46 schrieb Richard Bosch: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Denny, If you use mirrormaker to replicate data between clusters th

Re: mirror-maker

Hi Denny, If you use mirrormaker to replicate data between clusters then it needs access to all brokers. There is also a clear reason for this, Mirrormaker acts as a normal Kafka client when connecting to Kafka clusters. Mirrormaker uses the normal produce and consume protocol, which means that a

Re: mirror-maker

hi, it seems to be pretty clear .. TCPDump says, it tries to connect to the other hosts too, so I need to allow it. cu denny

Re: Mirror maker worker can't issue with REST uri

ffect of this is there is enormous amount of log generated. >> Is there a quick solution possible to slow down the logs. >> >> Cheers. >> >> From: Greg Harris >> Date: Wednesday, 8 February 2023 at 1:08 pm >> To: users@kafka.apache.org >> Subject: R

Re: Mirror maker worker can't issue with REST uri

k solution possible to slow down the logs. > > Cheers. > > From: Greg Harris > Date: Wednesday, 8 February 2023 at 1:08 pm > To: users@kafka.apache.org > Subject: Re: Mirror maker worker can't issue with REST uri > NetApp Security WARNING: This is an external email. Do

Re: Mirror maker worker can't issue with REST uri

Yes, that makes sense thanks. But the side effect of this is there is enormous amount of log generated. Is there a quick solution possible to slow down the logs. Cheers. From: Greg Harris Date: Wednesday, 8 February 2023 at 1:08 pm To: users@kafka.apache.org Subject: Re: Mirror maker worker

Re: Mirror maker worker can't issue with REST uri

Anup, This is the expected behavior of the MirrorMaker2 application when a connector attempts to reconfigure it's tasks. It is a limitation of the MirrorMaker2 distributed mode, and has an improvement in-progress that I don't believe has been released yet. See https://cwiki.apache.org/confluence/d

Re: Mirror Maker 2 with different Avro Schema Registries

hi, I guess this would mean a deserialisation error, meaning he data you're trying to read, is not as expected by the Confluent Avro deserializer. https://www.youtube.com/watch?v=GWBk9LIzy-A > On 29 Dec 2022, at 16:47, Дмитрий Лушников wrote: >

Re: Mirror maker disable auto topic creation

Thanks Ryanne Sent from Yahoo Mail for iPhone On Thursday, September 15, 2022, 12:00 AM, Ryanne Dolan wrote: The link is correct. Connect and mm2 create those internal topics at startup, whether or not auto topic creation is enabled. On Wed, Sep 14, 2022, 11:15 PM Mcs Vemuri wrote: > Hello,

Re: Mirror Maker is not replicating WRITE ACLs for topics and consumer group ACLs

Hello, this is a gentle reminder. Any response will be appreciated. John ср, 7 сент. 2022 г. в 15:03, John Lewis : > Hi everyone, > > I installed and ran Mirror Maker 2 to replicate all possible data from > Active cluster to Standby one. It's replicating topics, consumer groups, > topic READ AC

Re: Mirror maker disable auto topic creation

The link is correct. Connect and mm2 create those internal topics at startup, whether or not auto topic creation is enabled. On Wed, Sep 14, 2022, 11:15 PM Mcs Vemuri wrote: > Hello, > Is there any way to disable topic creation in MM2? I tried setting the > topic.creation.enable to False but the

Re: Mirror Maker 2 - High Throughput Identity Mirroring

Henry Cai has a PR somewhere. Ryanne On Wed, Mar 2, 2022, 3:36 AM Antón Rodríguez Yuste wrote: > Hi Ryanne, > > Is there a PR or code I could take a look at or just the KIP? "shallow > mirroring" seems very interesting for our use cases, and I would like to > evaluate it internally. > > Thanks,

Re: Mirror Maker 2 - High Throughput Identity Mirroring

Hi Ryanne, Is there a PR or code I could take a look at or just the KIP? "shallow mirroring" seems very interesting for our use cases, and I would like to evaluate it internally. Thanks, Antón On Thu, Jul 29, 2021 at 7:02 PM Ryanne Dolan wrote: > Jamie, this would depend on KIP-712 (or simila

Re: Mirror Maker Issue : Huge difference in topic size on disk

You could check whether the producers on the source cluster are using some compression algo. Use the same one on mm2 as currently it will decompress and then write again to the target cluster uncompressed unless you configure it. On Sun, Feb 6, 2022 at 05:20 Kafka Life wrote: > Dear Kafka Expert

Re: Mirror Maker 2: use prefix-less topic names?

Jake, in the most recent Kafka 3.0 release you will find IdentityReplicationPolicy, which does what you want. Just be careful that you don't try to replicate the same topics in a loop. Ryanne On Thu, Oct 7, 2021, 1:31 PM Jake Mayward wrote: > Hi, > > I have read a bit on MirrorMaker 2 via > htt

Re: Mirror Maker 2: use prefix-less topic names?

Nope. Mm2 has that prefix. I read that in 3.0, that prefix is going away. On Thu, Oct 7, 2021 at 11:31 AM Jake Mayward wrote: > Hi, > > I have read a bit on MirrorMaker 2 via > https://kafka.apache.org/documentation/#georeplication-overview, and I am > curious whether I can enable replication wi

Re: Mirror Maker 2 with different Avro Schema Registries

Quick informal update: Plugging in an SMT works like a charm 👍. Thanks for pointing me in the right direction. Took some time to understand that the MirrorSinkConnector mentioned in the KIP doesn't actually exist in Kafka. When running MM2 as a "pure" MirrorSourceConnector I need to point the conne

Re: Mirror Maker 2 with different Avro Schema Registries

Yes it's a consequence of Connect, which has its own serde model. mirror-makers hard-coded ByteArraySerialization for > consumers/producers

Re: Mirror Maker 2 with different Avro Schema Registries

Thanks Ryanne! I'll look into that. I was hoping for an "easier" solution, not having to write custom code for this. But that's probably where we'll end up. Does anyone know if mirror-makers hard-coded ByteArraySerialization for consumers/producers is by design? /Anders On Tue, Sep 14, 2021 at 3:5

Re: Mirror Maker 2 with different Avro Schema Registries

Hey Anders, take a look at Connect's serdes and SMTs. MirrorMaker can be configured to use them. Ryanne On Tue, Sep 14, 2021, 3:13 AM Anders Engström wrote: > Hi! > I'm trying to replicate a few topics using Mirror Maker 2 (2.8). > > Both the source and the target cluster use Schema Registry (K

Re: Mirror Maker 2 - High Throughput Identity Mirroring

Jamie, this would depend on KIP-712 (or similar) aka "shallow mirroring". This is a work in progress, but I'm optimistic it'll happen at some point. ftr, "IdentityReplicationPolicy" has landed for the upcoming release, tho "identity" in that context just means that topics aren't renamed. Ryanne

Re: Mirror Maker 2: Incoming messages on source and target kafka cluster mismatch after mirroring

Hi fighter, from the description, MM2 will not lose message if the MM2, source and target clusters are all in good heath. JMX metrics also rarely go significant wrong. My personal hint is: compare the aggregated network input and output of bandwidth of source and target cluster and see if match

Re: Mirror Maker 2: Incoming messages on source and target kafka cluster mismatch after mirroring

We migrated data from 7 node source cluster to target 9 node cluster. MM source connector producer does not have any compression as i mentioned earlier uses default producer. We use ByteArrayConverter for both producer key and value, idempotency =true. Is there a chance the JMX metrics can go wrong

Re: Mirror Maker 2 - Issues

Hi All, I was able to set up MM2 successfully but I see the same message being sent again and again. Also I see the following log message which I don't understand why. The topics on source and destination have the same number of partitions. Please give me some advice. [2021-03-10 09:41:02,072] IN

Re: Mirror Maker 2 - Issues

Thanks, I checked a consumer group using the below command on old and new clusters but the new cluster shows error consumer group doesn't exist. Am I missing something? As per the properties all groups should be included except the backlisted ones. kafka-consumer-groups.sh --bootstrap-server <> -

Re: Mirror Maker 2 - Issues

You can see the replicated consumer group offsets using the kafka-consumer-groups.sh tool. Make sure the consumer group is in the groups allowlist. Ryanne On Sun, Mar 7, 2021, 11:09 PM Navneeth Krishnan wrote: > Hi Ryanne, > > I commented out the alias config and set target to source as false.

Re: Mirror Maker 2 - Issues

Hi Ryanne, I commented out the alias config and set target to source as false. Now I don't see the error anymore and everything looks good. Thanks a lot for the feedback. One more question, how can I check if the consumer group offsets are replicated. If I switch over an application to this new cl

Re: Mirror Maker 2 - Issues

Navneeth, it looks like you are trying to override the cluster aliases to the empty string. If you look closely, these properties are defined using incorrect syntax. I'm not sure how the properties would be parsed, but you should fix that to rule it out as a source of confusion. Then, be aware tha

Re: Mirror Maker 2.0 NOT generating checkpoints for consumers running in assign mode

I am still facing the same issue. On Fri, Sep 11, 2020, 4:46 PM wrote: > Hi Ananya > Are you able to resolve this issue ,I'm also facing same issue . > > What parameter should be pass here if I'm doing failover from cluster A > ---> B > > Map newOffsets = > RemoteClusterUtils.transla

Re: Mirror Maker 2.0 NOT generating checkpoints for consumers running in assign mode

Hi Ananya Are you able to resolve this issue ,I'm also facing same issue . What parameter should be pass here if I'm doing failover from cluster A ---> B Map newOffsets = RemoteClusterUtils.translateOffsets(properties, "A", "TestTopic-123", Duration.ofMillis(5500)); Properties= Bo

Re: Mirror Maker 2.0 Queries

Thanks a lot Ryanne. That was really very helpful. On Thu, Aug 20, 2020, 11:49 PM Ryanne Dolan wrote: > > Can we configure tasks.max for each of these connectors separately? > > I don't believe that's currently possible. If you need fine-grained control > over each Connector like that, you might

Re: Mirror Maker 2.0 Queries

> Can we configure tasks.max for each of these connectors separately? I don't believe that's currently possible. If you need fine-grained control over each Connector like that, you might consider running MM2's Connectors manually on a bunch of Connect clusters. This requires more effort to set up,

Re: Mirror Maker 2.0 Queries

Thanks, Ryanne. That answers my questions. I was actually missing this "tasks.max" property. Thanks for pointing that out. Furthermore, as per the KIP of Mirror Maker 2.0, there are 3 types of connectors in a Mirror Maker Cluster: 1. KafkaSourceConnector - focus on replicating topic partitions

Re: Mirror Maker 2.0 Queries

Ananya, see responses below. > Can this number of workers be configured? The number of workers is not exactly configurable, but you can control it by spinning up drivers and using the '--clusters' flag. A driver instance without '--clusters' will run one worker for each A->B replication flow. So

Re: Mirror Maker 2.0 Queries

Any help here would be greatly appreciated. On Sat, Aug 8, 2020, 12:13 PM Ananya Sen wrote: > Thank you Ryanne for the quick response. > I further want to clarify a few points. > > The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka > connect we have multiple workers and each

Re: Mirror Maker 2.0 Queries

Thank you Ryanne for the quick response. I further want to clarify a few points. The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka connect we have multiple workers and each worker has some assigned task. To map this to Mirror Maker 2.0, A mirror Maker will driver have some

Re: Mirror Maker 2.0 Queries

Ananya, yes the driver is distributed, but each worker only communicates via kafka. They do not listen on any ports. Ryanne On Sat, Jul 11, 2020, 11:28 AM Ananya Sen wrote: > Hi > > I was exploring the Mirror maker 2.0. I read through this > > https://cwiki.apache.org/confluence/display/KAFKA/K

Re: Mirror Maker 2 MirrorClient

Hey Carl, that's what my team has done for our internal tooling, and I designed MirrorClient with that in mind. Given a single mm2.properties file you can create MirrorClients for each cluster and those in turn give you Admin/Consumer/Producer clients if you need them. Our internal tooling essentia

RE: Mirror Maker 2 internal topics

Hello Gérald, I have the exact same problem. Mirrormaker 2.0 Javadocs documentation is only slated for release 2.5.0 (see https://issues.apache.org/jira/browse/KAFKA-8930). I am also prototyping Mirrormaker 2.0 and I have successfully run the Mirrormaker 2.0 scripts (connect-mirror-maker.sh) u

RE: Mirror Maker: supplying broker properties with SSL password encrypted

Anyone has any suggestions?? From: Chandrajeet Padhy Sent: Monday, February 10, 2020 12:01 PM To: users@kafka.apache.org Subject: Mirror Maker: supplying broker properties with SSL password encrypted Is there a way to supply the consumer.config/producer.config property files having encrypted SS

Re: Mirror Maker 2 problems

Nils, a REST API for MM2 was discussed at length, but so far the community hasn't settled on a plan. For now, if you need to monitor MM2's Connectors over HTTP, you can use a bunch of Connect clusters and manually configure them to run MirrorSourceConnector etc. This has some advantages, e.g. you c

Re: Mirror Maker 2 problems

Another question is there anyway to open up a rest port like standard 8083 when running ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties have tried adding rest.port=8083 but does not seem to work. Would like to be able to query the status of the connector via rest. On Tue, J

Re: Mirror Maker 2 problems

Hello, yes that seems todo the trick. Another questions is there any documentation somewhere describing how the offset translations topics MM2 creates actually works? We need to write code ourself until this PR is merged I guess. https://github.com/apache/kafka/pull/7577 Thanks for the help. On

Re: Mirror Maker 2 problems

Hi Nils Can you try with the latest mm2 default config file found at https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties. Please feel free to reach out again if you are still stuck. Thanks Karan On Tue, Jan 14, 2020 at 1:48 PM Nils Hermansson < nils.hermans...@get

Re: Mirror Maker tool is not running

* > > what is type of parameters we have to use either in source side or else in > producer side ??? > > > > > > Please tell me due to this we are not getting data from DC to DR cluster. > > Please help to resolve this , Thanks a lot > > >

RE: Mirror Maker tool is not running

_ From: SenthilKumar K Sent: Tuesday, May 7, 2019 1:13:53 PM To: ASHOK MACHERLA Cc: users@kafka.apache.org; Ryanne Dolan Subject: Re: Mirror Maker tool is not running Looks like you are hitting : https://issues.apache.org/jira/browse/KAFKA-6947 or https://jira.apache.org/jira/browse/KAFKA-6177 --Senth

Re: Mirror Maker tool is not running

** > > what is type of parameters we have to use either in source side or else in > producer side ??? > > > > > > Please tell me due to this we are not getting data from DC to DR cluster. &

RE: Mirror Maker tool is not running

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: ASHOK MACHERLA Sent: Sunday, May 5, 2019 11:19:48 PM To: users@kafka.apache.org Subject: Re: Mirror Maker tool is not running Dear Ryanne Where can I use that compr

Re: Mirror Maker tool is not running

Dear Ryanne Where can I use that compression method either source side or target side??? What type of parameters can I required ?? What is the configuration ??? Can please tell Ryanne. Thanks for your help.. Sent from Outlook

Re: Mirror Maker tool is not running

This can be caused by compressed messages being decompressed by the MirrorMaker consumer but not recompressed by the producer. In order to prevent this, enable shallow iteration in the consumer, or enable compression in the producer. Ryanne On Sun, May 5, 2019, 8:35 AM ASHOK MACHERLA wrote: > H

Re: Mirror Maker tool is not running

Hello This is new tool and our client doesn't give permission to use this tool. Currently mirror maker is working fine. But in server.properties DC cluster we set 75KB in message.max.bytes=75000 DR cluster we set 1.5 MB message.max.bytes=150 Here , source is DC, target is DR, Data going

Re: Mirror Maker tool is not running

Hi, I have written a tool like mirror maker: https://github.com/mykidong/kafka-spiegel/blob/master/README.md You can try this. Cheers, Kidong Lee 2019년 5월 4일 (토) 오후 8:38, ASHOK MACHERLA 님이 작성: > > > Sent from Outlook > > On 30-Apr-2019 3:37 PM, ASHOK MACHERLA wrote: > > Dear Team > > > > Pl

Re: Mirror Maker tool is not running

Sent from Outlook On 30-Apr-2019 3:37 PM, ASHOK MACHERLA wrote: Dear Team Please help us , our mirror maker tool is not running properly Please look into this belowmirror maker log file exceptions ** [2019-03-11 18:3

RE: Mirror Maker Errors

Hi Oleg, do you have configured your consumer/producer with "no data loss" configuration like bellow ? For Consumer, set auto.commit.enabled=false in consumer.properties For Producer 1. max.in.flight.requests.per.connection=1 2. retries=Int.MaxValue 3. acks=-1 4. block.on.buffer.f

Re: mirror maker producer thread dies with error

"Memory records is not writable" error was fixed in 0.10.0.0 release https://issues.apache.org/jira/browse/KAFKA-3594 On Fri, Jan 12, 2018 at 6:10 AM, Sunil Parmar wrote: > We see multiple instance of this error > > 2017-12-23 05:30:53,722 WARN > org.apache.kafka.clients.producer.internals.Sende

Re: Mirror maker Configs 0.9.0

Hello Experts, I am trying to replicate data between On Prem Kafka Cluster(Source) and another Kafka cluster (Target?) set up in a Cloud provider environment. The On-Prem to cloud is connected via IPSec VPN and Mirror maker tool is used on 0.10.2.x version. The consumer configs are as fo

Re: Mirror Maker - Message Format Issue?

Hi Craig, I think you may be running into: https://issues.apache.org/jira/browse/KAFKA-4073 Ismael On Thu, Oct 13, 2016 at 5:51 AM, Craig Swift < craig.sw...@returnpath.com.invalid> wrote: > Hello, > > Just to close this issue out. The 8 producer going to the 10 cluster was > the root issue. T

Re: Mirror Maker - Message Format Issue?

Hello, Just to close this issue out. The 8 producer going to the 10 cluster was the root issue. The mirror maker by default was unable to produce the message to the destination cluster. The work around was to include a MirrorMakerMessageHandler that did nothing but repackage the message again. In

Re: Mirror maker higher offset in the mirror.

Things like consumer rebalances on the cluster you copy from, and brokers going down on the cluster your writing down can cause duplications. The default settings are set to prevent data loss, making data duplication more likely to happen in case of error. You could possibly make a simple consumer

Re: Mirror maker setup - multi node

Then it's much simpler, just use replication-factor=3 for your topics, either by setting it when creating topics manually, or by setting default.replication.factor=3 in the broker settings. Kafka will then take care of the leadership on the partitions so both the load is balanced, and in case some

Re: Mirror maker setup - multi node

Hi there, I mean 1 cluster with 3 nodes. So I will need to run the mirror maker cluster on each of the 3 nodes in the cluster, in case of the loss of a node, the other 2 will continue to pull messages off the consumer cluster. It does seem to work correctly when I tested it. It just warns about to

Re: Mirror maker setup - multi node

With 3 nodes, I assume you mean 3 clusters? If I understand correctly, say you have 3 clusters, A, B, and C, you simultaneously: - want to copy from A and B to C, to get an aggregation in C - want to copy fram A and C to B, to get a fail-back aggregation in B. Now what will happen when a message is

RE: Mirror maker Configs 0.9.0

Thanks a lot Stephen.. It worked:) -Original Message- From: Stephen Powis [mailto:spo...@salesforce.com] Sent: Wednesday, March 09, 2016 6:58 PM To: users@kafka.apache.org Cc: Kishore N R Subject: Re: Mirror maker Configs 0.9.0 I've attached my two configs here. Pay close atte

Re: Mirror maker Configs 0.9.0

I've attached my two configs here. Pay close attention to the --num-streams argument to mirror-maker. I have a lot of thru-put on my topics so I ended up matching the number of streams = number of partitions for each of my topics. A stream is essentially just a consumer and producer thread. If

Re: Mirror maker Configs 0.9.0

Thanks for the reply.. I will remove the bootstrap.servers property and add zookeeper.connect in consumer properties and let you know Also, is there any way we can check how much data the target data center is lagging behind source DC? On Wed, Mar 9, 2016 at 3:41 PM, Gerard Klijs wrote: >

Re: Mirror maker Configs 0.9.0

What do you see in the logs? It could be it goes wrong because you have the bootstrap.servers property which is not supported for the old consumer. On Wed, Mar 9, 2016 at 11:05 AM Gerard Klijs wrote: > Don't know the actual question, it matters what you want to do. > Just watch out trying to cop

Re: Mirror maker Configs 0.9.0

Don't know the actual question, it matters what you want to do. Just watch out trying to copy every topic using a new consumer, cause then internal topics are copied, leading to errors. Here is a temple start script we used: #!/usr/bin/env bash export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote

Re: Mirror maker Configs 0.9.0

Hi Experts, I am trying to replicate data between different data centers using mirror maker tool. kafka-run-class.bat kafka.tools.MirrorMaker --consumer.config consumer.properties --producer.config producer.properties --whitelist * Can someone provide the sample consumer.properties and producer.

Re: mirror maker against 0.8.2 source cluster and 0.9.0 destination cluster

The MirrorMaker 0.9.0 has some nice features like messageHandler which would be great to use. Any ideas how this new functionality could be used in a 0.8.2 source cluster and 0.9.0 destination cluster in an efficient way? Thanks On Wed, Jan 6, 2016 at 11:30 AM, Ismael Juma wrote: > Hi Stephen,

Re: mirror maker against 0.8.2 source cluster and 0.9.0 destination cluster

Hi Stephen, Newer brokers support older clients, but not the other way around. You could try 0.8.2 MirrorMaker against 0.8.2 source and 0.9.0 target clusters perhaps? Ismael On 6 Jan 2016 11:18, "Stephen Powis" wrote: > Hey! > > So I'm trying to get mirror maker going between two different clus

Re: Mirror maker SSL support

Probably within 2-3 weeks. Thanks, Jun On Mon, Aug 31, 2015 at 10:54 AM, Xiang Zhou (Samuel) wrote: > Thank you for update Jun. > What's the schedule to get new consumer implemented in Mirror Maker? > > Samuel > > On Mon, Aug 31, 2015 at 10:06 AM, Jun Rao wrote: > > > Currently, MirrorMaker c

Re: Mirror maker SSL support

Thank you for update Jun. What's the schedule to get new consumer implemented in Mirror Maker? Samuel On Mon, Aug 31, 2015 at 10:06 AM, Jun Rao wrote: > Currently, MirrorMaker can run the producer with SSL. We will add the > option to run the new consumer in MirrorMaker in KAFKA-2452, which wil

Re: Mirror maker SSL support

Currently, MirrorMaker can run the producer with SSL. We will add the option to run the new consumer in MirrorMaker in KAFKA-2452, which will enable SSL on the consumer side too. Thanks, Jun On Fri, Aug 28, 2015 at 12:01 PM, Xiang Zhou (Samuel) wrote: > Hi, folks, > > I just want to check if a

Re: Mirror maker doesn't rebalance after getting ZkNoNodeException

I use commit 9e894aa0173b14d64a900bcf780d6b7809368384 from trunk code On Wed, 10 Jun 2015 at 01:09 Jiangjie Qin wrote: > Which version of MM are you running? > > On 6/9/15, 4:49 AM, "tao xiao" wrote: > > >Hi, > > > >I have two mirror makers A and B both subscripting to the same whitelist. > >Du

Re: Mirror maker doesn't rebalance after getting ZkNoNodeException

Which version of MM are you running? On 6/9/15, 4:49 AM, "tao xiao" wrote: >Hi, > >I have two mirror makers A and B both subscripting to the same whitelist. >During topic rebalancing one of the mirror maker A encountered >ZkNoNodeException and then stopped all connections. but mirror maker B >di

Re: Mirror maker fetcher thread unexpectedly stopped

Thank you for the explanation. Patch submitted https://issues.apache.org/jira/browse/KAFKA-2048 On Wed, Mar 25, 2015 at 8:29 AM, Jiangjie Qin wrote: > It should be another ticket. This is a AbstractFetcherThread issue rather > than a mirror maker issue. > > I kind of think this case you saw was

Re: Mirror maker fetcher thread unexpectedly stopped

It should be another ticket. This is a AbstractFetcherThread issue rather than a mirror maker issue. I kind of think this case you saw was a special case as it¹s not actually a runtime error but a coding bug. Fetcher thread should not die by design. So I don¹t think we have a way to restart fetche

Re: Mirror maker fetcher thread unexpectedly stopped

The other question I have is the fact that consumer client is unaware of the health status of underlying fetcher thread. If the fetcher thread dies like the case I encountered is there a way that consumer can restart the fetcher thread or release ownership of partitions so that other consumers can

Re: Mirror maker fetcher thread unexpectedly stopped

Thanks JIanjie. Can I reuse KAFKA-1997 or should I create a new ticket? On Wed, Mar 25, 2015 at 7:58 AM, Jiangjie Qin wrote: > Hi Xiao, > > I think the fix for IllegalStateExcepetion is correct. > Can you also create a ticket and submit a patch? > > Thanks. > > Jiangjie (Becket) Qin > > On 3/24/

  1   2   >