Re: Apache Kafka cluster to cluster

2020-04-29 Thread Manoj.Agrawal2
Is there documentation or example for mirror maker 2.0 ? On 4/29/20, 9:04 PM, "Liam Clarke-Hutchinson" wrote: [External] Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is

Re: Apache Kafka cluster to cluster

2020-04-29 Thread vishnu murali
Hi Clark My source cluster is on localhost:9092 My Target cluster was localhost:9091 I am not aware of something like Mirror maker 1 and 2 Because I am using mirror maker which come default from Apache Kafka 2.3.0 On Thu, Apr 30, 2020, 09:43 Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz>

Re: Apache Kafka cluster to cluster

2020-04-29 Thread Liam Clarke-Hutchinson
Hi Vishnu, I can see you're using Mirror Maker 1, unlike Mirror Maker 2, this doesn't auto-create topics on the target cluster. Does your target cluster on port 9091 have the same topics created on it that exist on the source cluster on port 9092? Also note that a Kafka consumer defaults to "late

Re: Apache Kafka cluster to cluster

2020-04-29 Thread Liam Clarke-Hutchinson
Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror Maker 1.0. On Thu, Apr 30, 2020 at 6:51 AM Blake Miller wrote: > Oh, and it looks like Confluent has released a newer replacement for >

Re: Apache Kafka cluster to cluster

2020-04-29 Thread vishnu murali
i start the kafka mirror using the below configuration .\bin\windows\kafka-mirror-maker.bat --consumer.config .\config\consumer.properties --producer.config .\config\producer.properties --whitelist=".*" *Consumer.properties:* # format: host1:port1,host2:port2 ... bootstrap.servers=localhost:9092

Re: Apache Kafka cluster to cluster

2020-04-29 Thread vishnu murali
Thanks Blake..😊😊 More over can we use any connector types for this situation? Like source is a topic and sink is also an another topic Is this possible... On Thu, Apr 30, 2020, 00:19 Blake Miller wrote: > Hi Vishnu, > > Check out MirrorMaker > https://cwiki.apache.org/confluence/pages/viewpag

Re: Apache Kafka cluster to cluster

2020-04-29 Thread Blake Miller
Oh, and it looks like Confluent has released a newer replacement for MirrorMaker called Replicator https://docs.confluent.io/current/multi-dc-deployments/replicator/migrate-replicator.html On Wed, Apr 29, 2020 at 6:49 PM Blake Miller wrote: > Hi Vishnu, > > Check out MirrorMaker > https://cwi

Re: Apache Kafka cluster to cluster

2020-04-29 Thread Blake Miller
Hi Vishnu, Check out MirrorMaker https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 This can do what you want. Note that the offsets are not copied, nor are the message timestamps. HTH On Wed, Apr 29, 2020 at 6:47 PM vishnu murali wrote: > Hi Guys, > > I am having two

Apache Kafka cluster to cluster

2020-04-29 Thread vishnu murali
Hi Guys, I am having two separate Kafka cluster running in two independent zookeeper I need to send a set of data from one topic from cluster A to cluster B with the same topic name with all data also.. How can I achieve this Done anyone have any idea ??