MirrorMaker2 to mirror new topics?

Hi, Should MM2 automatically start to mirror new topics? I see in the logs that MM2 created the remote topic, but it is not mirroring the new messages. In the source cluster, topic was created 3 days ago and there is about 200 million messages in it. Just to test if the messages are there, I coul

Re: MirrorMaker2 to mirror new topics?

Hi, it is mirroring but the problem was that on of the task failed with not being able to produce messages to the destination cluster and the task stopped there. Peter On Mon, 9 Mar 2020 at 11:20, Péter Sinóros-Szabó < peter.sinoros-sz...@transferwise.com> wrote: > Hi, > > Should MM2 automatica

Question about log flusher real frequency

Hello, By default, both log.flush.interval.ms and log.flush.interval.messages are set to Long.MAX_VALUE. As I understand, it makes Kafka flush log to disk (fsync) only depends on file system. Is there any simple way to monitor that frequency ? Is there a rule of thumb to estimate that value dep

Re: Question about log flusher real frequency

Hi Alexandre, Thank you for your quick answer. I want to monitor it cause I'm trying to find out the reason why our existing Kafka cluster is configured to flush data every10 milliseconds! (people who configured it are not available anymore to answer). As that value seems really low to me, I was

Re: MirrorMaker2 to mirror new topics?

Hey Peter, often the problem is the replication factor or min ISRs being misconfigured, which can prevent MM2 (or anything else) from sending. Hope that helps! Ryanne On Mon, Mar 9, 2020, 10:35 AM Péter Sinóros-Szabó wrote: > Hi, > > it is mirroring but the problem was that on of the task faile

Re: Passing states stores around

Hi All, Any suggestions? Thanks On Sat, Mar 7, 2020 at 10:13 AM Navneeth Krishnan wrote: > Hi All, > > Is there a recommended way of passing state stores around across different > classes? The problem is state store can be fetched only if you have access > to the context and in most of the sce

Re: Subject: [VOTE] 2.4.1 RC0

Thanks to everyone for voting. A new blocker has surfaced https://issues.apache.org/jira/browse/KAFKA-9675, so I'll do another RC soon. Thanks again. Bill On Mon, Mar 9, 2020 at 1:35 PM Levani Kokhreidze wrote: > +1 non-binding. > > - Built from source > - Ran unit tests. All passed. > - Quick

Re: Subject: [VOTE] 2.4.1 RC0

Is this a blocker given that it's been like this for months and no-one noticed? 2.4.1 seemingly has all the votes needed for the release. Why not go ahead with it. When KAFKA-9675 is merged, it can be included in the next release. Ismael On Mon, Mar 9, 2020 at 8:43 PM Bill Bejeck wrote: > Thank

Re: Passing states stores around

Hi Navneeth, This sounds like an unusual use case. Can you provide more information on why this is required? Thanks, John On Mon, Mar 9, 2020, at 12:48, Navneeth Krishnan wrote: > Hi All, > > Any suggestions? > > Thanks > > On Sat, Mar 7, 2020 at 10:13 AM Navneeth Krishnan > wrote: > > > H

[VOTE] 2.5.0 RC1

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 2.5.0. The first release candidate included an erroneous NOTICE file, so another RC was needed to fix that. This is a major release of Kafka which includes many new features, improvements

Re: Passing states stores around

Hi John, I'm using PAPI to create my topology which has 5 process functions. Out which 3 are large functions (more than 1000 lines of code) and they have about 2 KV stores each. Since the code is fairly large per function, I have them split into classes by functionalities. and some method in the c

Re: Passing states stores around

Well I can suggest is that you can access state store via processor's context. Once you have the reference of the state store it can be passed to different classes via the same reference. On Tue, Mar 10, 2020 at 8:57 AM Navneeth Krishnan wrote: > Hi John, > > I'm using PAPI to create my topolog