Hi fighter,
from the description, MM2 will not lose message if the MM2, source and target
clusters are all in good heath. JMX metrics also rarely go significant wrong.
My personal hint is: compare the aggregated network input and output of
bandwidth of source and target cluster and see if match
> > Content-Type: application/json
> > > Content-Length: 1251
> > > Expect: 100-continue
> > >
> > < HTTP/1.1 100 Continue
> > * We are completely uploaded and fine
> > < HTTP/1.1 405 Method Not Allowed
> > < Date: Wed, 14 Apr 2021 19:07:
correct?
>
> thanks,
>
>
>
> On Fri, Apr 9, 2021 at 7:55 AM Ning Zhang wrote:
>
> > Hi Men,
> >
> > I used to deploy MM2 on EC2 with SSL and IIRC, probably give a try of
> > self-signing certs and key for testing purpose:
> > https://linuxize.co
Hi Men,
I used to deploy MM2 on EC2 with SSL and IIRC, probably give a try of
self-signing certs and key for testing purpose:
https://linuxize.com/post/creating-a-self-signed-ssl-certificate/
On 2021/04/09 03:14:30, Men Lim wrote:
> Hi Ryanne,
>
> thanks for the reply. My kafka clusters are
mpty, not to the source
> CG offset.
>
> The negative lag can cause CGs to miss messages during a migration if new
> messages are sent between stopping a CG on the source cluster and starting
> the CG on the target cluster.
>
>
> On Fri, Mar 26, 2021 at 1:52 AM Ning Zha
:10:51.386] INFO [Consumer clientId= consumer-4,
> > groupId=null] Seeking to offset 52 for partition
> > ONPREM.AWS.REPLICA.TOPIC.P3R3-0
> > (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> >
> >
> >
> >
> > You can s
Hello Frank,
Thanks for helping on analyzing the issue.
Regarding when the CG offsets at destination cluster will be updated. From the
current codebase, there seems 2 criteria:
(1) if the CG offsets do not contain a pair of , simply sync
the offsets from source
(2) for a pair of , if the conve
aker2 does not guarantee exactly-once delivery but
> is there a way to maybe sync to topics so the kafka connect will be aware of
> each other offset.
>
> Daniel
>
> > On 19 Mar 2021, at 9:09, Ning Zhang wrote:
> > Hi Daniel, MirrorMaker2 creates its own "offset
just my 2 cents
the best answer is always from the real-world practices :)
RocksDB https://rocksdb.org/ is the implementation of "state store" in Kafka
Stream and it is an "embedded" kv store (which is diff than the distributed kv
store). The "state store" in Kafka Stream is also backed up by "
Hi Daniel, MirrorMaker2 creates its own "offsets" topic to track the process of
consumption.
just my 2 cents - If you already have two Kafka connect clusters in two
different sites, it sounds practical to:
(1) use "cluster" mode, instead of "dedicated" mode of MirrorMaker2
(2) add one "MirrorMak
gt; where is mentioned the “at-least” delivery guarantee? Just for the record.
>
> Kind Regards,
>
> Από: Ning Zhang
> Αποστολή: Τετάρτη, 17 Μαρτίου 2021 22:39
> Προς: users@kafka.apache.org
> Θέμα: Re: Mirrormaker 2.0 - duplicates with idempotence enabled
>
> Hello Vang
Hello Vangelis,
By default, current MM 2.0 guarantees "at-least" once delivery guarantee,
meaning there will be duplicate messages under some failure scenarios.
If you prefer to no-message loss, there is a pending PR about MM 2.0
https://issues.apache.org/jira/browse/KAFKA-10339
On 2021/03/10
Hello Alan,
I may probably see the similar case. One quick validation that could be run is
to test on the source cluster with higher Kafka version. If still not working,
please email me and I could introduce you to person who may have similar case
before.
On 2021/03/15 21:59:03, Alan Ning wro
entirely consuming the messages at the original
> cluster.
> This commit illustrates this situation.
> https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7
>
> regards, aki
>
> El mié, 13 ene 2021 a las 3:17, Ning Zhang ()
> escribió:
> >
>
Hello Aki,
Can you elaborate on "when there are some messages left in the original
cluster" ?
On 2021/01/12 13:01:40, Aki Yoshida wrote:
> Hi Ryanne,
> Thanks for the information regarding the manual translation approach.
> But for 2.7.0, I have a question. I thought this translation would
>
e wrote:
> In case we opt to choose some third party store instead of kafka's stores
> for storing state (e.g. Redis cache or Ignite), then will we lose the
> exactly-once guarantee provided by kafka and the state stores can be in an
> inconsistent state ?
>
> On Sa
The physical store behind "state store" is change-log kafka topic. In Kafka
stream, if something fails in the middle, the "state store" is restored back to
the state before the event happens at the first step / beginning of the stream.
On 2020/12/31 08:48:16, Pushkar Deole wrote:
> Hi All,
Hi YG, it seems current Kafka Connect already provided simple REST endpoint for
connector and task level status (running / stopped / ):
https://docs.confluent.io/5.5.0/connect/managing/monitoring.html
Is there any other info about Kafka Connect you care about?
On 2020/12/31 19:36:38, YG Che
ic":"TEST"}]
> Value: {"offset":24}
>
> After posting the message, once the mirror maker is restarted, it will read
> the internal topic to get the latest offset of that topic for which the
> message has to be replicated and this way we can ensure no duplic
As org.kaliy.kafka.connect.rss.RssSourceConnector is open-source, it seems a
better idea to extend it to read from multiple RSS URI, so you may still enjoy
the fine-granularity of automated management of a set of URI within one
connector, while not creating too many connectors
On 2020/12/09 16:
e that groupId is still null in MM2 and the offsets are previous
> offset meaning it will replicate those messages as well which has been
> already replicated by Replicator.
>
>
>
> Thanks and regards,
>
> Amit
>
>
>
> -Original Message-
One crude way we found and is in testing is to manually change the offset of
> the topic in the internal topics which MM2 reads to get the latest offset of
> the message.
>
> Thanks,
> Amit
>
> -Original Message-
> From: Ning Zhang
> Sent: Monday,
Hi Amit, I guess you may need to override the actual consumer group config
(probably not consumer.group.id) that is used in Kafka Connect
On 2020/11/26 06:47:11, wrote:
> Hi All,
>
> We are currently trying to migrate Confluent replicator to Apache Open Source
> Mirror Maker v2.0. We are fac
if your new topics are not named "topic1" or "topic2", maybe you want to use
regex * to allow more topics to be considered by Mm2
# regex which defines which topics gets replicated. For eg "foo-.*"
src-cluster->dst-cluster.topics = topic1,topic2
On 2020/10/30 01:48:00, "Devaki, Srinivas" wrote:
Here my initial thoughts:
(1) MM2 assigned partitions over tasks based on round-robin at the MM2
application level:
https://github.com/apache/kafka/commit/4fe324b5a696d47ae8414cf030ae36b83a2cf163#diff-bf03ad7a93c070fe11fcf88f897e0973
(2) if want to try out `RoundRobinAssignor` in consumer-level
though. The
> config is rendered fine so I need to dig deeper on my side as this is
> working for you.
>
> Thanks,
>
> On Tue, Sep 15, 2020 at 5:29 PM Ning Zhang wrote:
>
> > Hello Samuel,
> >
> > I tried other consumer config and it worked. Basically
/15 20:53:12, Samuel Cantero wrote:
> I tried this but even the override does not work for me as I can still see
> "earliest" on the ConsumerConfig properties. Did you manage actually to
> make this work or are you just recommending this config?
>
> Best,
>
> On Mo
That is possible, please check out this thread:
https://lists.apache.org/thread.html/r978775b2dbdf34a5c6c40105e2e69ad205fa94e33cef08e8519389c9%40%3Cusers.kafka.apache.org%3E
On 2020/09/02 07:10:45, Iftach Ben-Yosef wrote:
> Hello,
>
> Whenever we add a new topic to the mirroring whitelist it s
Hello Samuel,
I guess you are talking about the case where you start to mirror a new topic?
If yes, I think the config should be
`source-cluster-alias.consumer.auto.offset.reset`, instead of `target`, because
`target` is where the producer of MM2 replicates the messages towards, right?
If MM2
29 matches
Mail list logo
