Question about Mirror Maker 2 connector

ansforms.RegexRouter transforms.RenameTopicSimple.regex=.* transforms.RenameTopicSimple.repalcement=target-kafka-topic-name ``` My question is: 1. Can *configurations above* + *MirrorMaker 2 source connector *meet our requirements? 2. Is there any potential issues when messages from multiple topics

Re: Question about replication mechanism and leader election in Kafka

> but it’s not directly tied to a quorum mechanism Correct. > this doesn’t really differ from having only three brokers Also correct. However, practically, we often reassign failed replicas to another broker (e.g. using CruiseControl) in case of broker crash so having 3 + 1 brokers is more safe.

Re: Question about replication mechanism and leader election in Kafka

Thank you very much, Haruki! This really clarifies things for me. I just want to confirm one more point: it seems that a majority quorum in Kafka doesn’t actually play a role in leader election (it might make the cluster more reliable overall, but it’s not directly tied to a quorum mechanism). Am

Re: Question about replication mechanism and leader election in Kafka

Hi Ilya, > Does it require some sort of majoritary quorum to decide whose offsets are valid No. Controller just elects a new leader from ISR (in current Kafka version, the leftmost ISR will be chosen though) without taking offsets into account. After an ISR becomes a new leader, other replicas tr

Question about replication mechanism and leader election in Kafka

Hello, I’m trying to understand how replication and leader election work in Kafka, and I have a couple of questions. I know that Kafka does not rely on a majority-quorum approach. Instead, there is an ISR set, which consists of replicas that the current leader considers sufficiently up-to-date (t

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

KIP-950 > >> > < > >> > > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D950-253A-2B-2BTiered-2BStorage-2BDisablement-23KIP950-3ATieredStorageDisablement-2DDisablement-2DZookeeperBackedCluster&d=

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

ch > > describe the disablement of kafka tiered storage. It said that zk mode > > won't be supported and I can't find out the reason, with confusion, I > have > > a question: > > This KIP proposes an implement with stopReplicas v5 to let brokers know > > which

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

hank you. >> Luke >> >> On Mon, Oct 28, 2024 at 12:32 PM Jianfeng Lin >> wrote: >> >> > Hi kafka community, I read the KIP-950 >> > < >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

t; > > > > which > > describe the disablement of kafka tiered storage. It said that zk mode > > won't be supported and I can't find out the reason, with confusion, I > have > > a question: > > This KIP proposes an implement with stopReplicas v5 to le

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

kafka tiered storage. It said that zk mode > won't be supported and I can't find out the reason, with confusion, I have > a question: > This KIP proposes an implement with stopReplicas v5 to let brokers know > which topic partitions should stop their remote storage functions when

Kafka Tiered Storage Disablement Implement in ZK Mode Question

pported and I can't find out the reason, with confusion, I have a question: This KIP proposes an implement with stopReplicas v5 to let brokers know which topic partitions should stop their remote storage functions when "remote.log.copy.disable" is set to true, different from kraft m

Re: MirrorMaker2 Question

Hi Harry, Thanks for your question! There were negative lag/data loss bugs fixed some time back [1, 2] so check what version of the MirrorMaker connectors you're using. I would recommend using the latest version of MM2 (3.8.0) if possible. After upgrading you will need to clear the checkp

MirrorMaker2 Question

Hi everyone, Does anyone know if it is possible for the Checkpoint connector to be ahead of the Source connector? For context, I have a target Kafka cluster that is reporting negative lag because the log end offset is 0 but the consumer group offset is 100. I have a tenuous theory that this co

Re: Question about config.action.reload

Hi Ran, The config.action.reload, subscribe, unsubscribe, and ConfigChangeCallback features are partially-implemented. As far as I can tell if you're using only AK connect-runtime and config providers, none of it is functional. If you have a config provider that implements TTLs, the runtime will

Question about config.action.reload

Hi Kafka team, Currently I'm investigating the config.action.reload feature of Kafka Connect. I'm wondering how to enable this feature. E.g. If I use the FileConfigProvider, clients/src/main/java/org/apache/kafka/common/config/provider/FileConfigProvider.java is it enough to set the config.action.

Question about moving kafka brokers between AZs in AWS

Hello, Here is the scenario. We have a kafka/zookeeper ensemble, where kafka and zookeeper are installed on the same node (instance). There are three zk nodes and two kafka nodes. In other words, out of the three zk nodes, kafka is present in two. Now, these two kafka brokers are in different AZ

Question

I have a consumer group that consumes a lot of topics, he suddenly stops consuming a certain topic, but this topic is consumed by other consumer groups, how do I get this consumer group to unsubscribe from this topic?

question: mirror maker 2

Hello all, I am setting up mm2 to replicate messages, consumer groups, and consumer offset from a->b. I believe I am replicating those 3 items from a->b. my mm2 prop file is as followed: ``` # specify any number of cluster aliases clusters = a,b b.group.id=mm2-request # replication settings ta

Re: kafka cluster question

Hi, As Artem mentioned, I did some tests with setting replication factor 1 and 3 for two different topics One of the kafka broker is down: The command works if the replication factor is 3. (*testtopicreplica3 is created with rf 3)* *[root@node-223 kafka_2.12-2.8.2]# ./bin/kafka-consumer-gr

Re: kafka cluster question

Hi, Just a long shoot, but I might be wrong. You have offsets.topic.replication.factor=1 in you config, when one broker is down, some partitions of __consumer_offsets topic will be down either. So kafka-consumer-groups can't get offsets from it. Maybe it's just a little misleading error message.

Re: kafka cluster question

Hi, sorry for the confusion, here is details: I have 3 broker nodes: 192.168.20.223 / 224 / 225 When all kafka services are UP: [image: image.png] I stopped the kafka service on *node 225*: [image: image.png] Then i tried the command on node223 with --bootstrap-server 192.168.20.223:9092,192.16

Re: kafka cluster question

Hi. Which server did you shutdown in testing? If it was 192.168.20.223, that is natural kafka-consumer-groups script fails because you passed only 192.168.20.223 to the bootstrap-server arg. In HA setup, you have to pass multiple brokers (as the comma separated string) to bootstrap-server so that

kafka cluster question

Hi all, I'm trying to do some tests about high availability on kafka v2.8.2 I have 3 kafka brokers and 3 zookeeper instances. when i shutdown one of the kafka service only in one server i got this error: [root@node-223 ~]# /root/kafka_2.12-2.8.2/bin/kafka-consumer-groups.sh --bootstrap-server 192

Re: MM2 Question

Question Hi Vinay, I am sorry to hear about your difficulties with MirrorMaker2. Are you using the MirrorMaker2 dedicated mode, or do you have the MirrorMaker2 connectors running in a separate Connect cluster? If you're using the dedicated mode, that is a known problem on <3.5 an

Re: MM2 Question

Hi Vinay, I am sorry to hear about your difficulties with MirrorMaker2. Are you using the MirrorMaker2 dedicated mode, or do you have the MirrorMaker2 connectors running in a separate Connect cluster? If you're using the dedicated mode, that is a known problem on <3.5 and is the motivation for ht

MM2 Question

Hi Team, I am fairly new to Kafka but one issue that we are currently battling is in MM2. MM2 for no reason is missing new topics that are present in one of the source clusters (no obvious error message in MM2 error logs). I heard from our internal team that this MM2 has 10k topics to take care

Re: Question about using public classes from the internal packages

;> >> [1] >> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals >> >> On 12/18/23 1:46 PM, John Brackin wrote: >> > Hi all, >> > >> > I have a question about using classes that are are public but the class

Re: Question about using public classes from the internal packages

considered public API [1]. For public APIs we guarantee backwards > compatibility. > > Best, > Bruno > > [1] > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > On 12/18/23 1:46 PM, John Brackin wrote: > > Hi all, > > >

Re: Question about using public classes from the internal packages

. Best, Bruno [1] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals On 12/18/23 1:46 PM, John Brackin wrote: Hi all, I have a question about using classes that are are public but the class is contained in the internals packages Generally, would it be supported if I

Question about using public classes from the internal packages

Hi all, I have a question about using classes that are are public but the class is contained in the internals packages Generally, would it be supported if I wrote Kafka code referencing these classes? This question comes from an attempt to add a CacheListener to a TimestampedKeyValueStore. The

Re: [Question] About Kafka producer design decision making

Hi. I also guess the main reason for using Future was for JDK1.7 support which is no longer necessary in the current Kafka version. Actually, there's a KIP about this: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459 but it seems it's not active now. > I wonder if it is

Fwd: [Question] About Kafka producer design decision making

Dear Apache Kakfa Developers, I'm 4-year SWE in South Korea. I have some questions while watching Kafka Producer API. *Why Use "Future" and Not "CompletableFuture"?* In the case of "Future", blocking occurs when calling "*get()*", so I thought "Computable Future" would be better when doing more

Question regarding controller election problems on Kafka 3.5.0 (protocol 3.5)

Hello all, I have recently upgraded a number of Kafka clusters from Kafka 2.5.1 (protocol 2.5) to Kafka 3.5.0 (protocol 3.5) according to this steps outlined here: https://kafka.apache.org/documentation/#upgrade_350_zk For the majority of these clusters, the new version and protocol have been runn

kafka question

Hi, I encountered a problem and I don't know if it's a bug. The description is as follows, kafka version 3.3.2 with kraft 1. one topic has two replica 0 and 1, 0 is leader and 1 is follower 2. at some point, the disk on the broker id 0 is error(Read-only file system), but log not offline in th

RE: Kafka Connect Rest Extension Question

ge- From: mil...@votecgroup.com [mailto:mil...@votecgroup.com] Sent: 01 August 2023 11:56 To: users@kafka.apache.org Subject: RE: Kafka Connect Rest Extension Question Hi Team, Greetings, We actually reached out to you for Oracle/ IT / SAP / Infor / Microsoft "VOTEC IT SERVICE PARTNERSH

RE: Kafka Connect Rest Extension Question

e- From: 양형욱 [mailto:hyungwooky...@navercorp.com] Sent: 31 July 2023 14:42 To: users@kafka.apache.org Subject: Kafka Connect Rest Extension Question https://stackoverflow.com/questions/76797743/how-can-i-solve-connectrestextension-error There is an issue with this link where the ConnectR

RE: Kafka Connect Rest Extension Question

inal Message- From: Greg Harris [mailto:greg.har...@aiven.io.INVALID] Sent: 31 July 2023 23:42 To: users@kafka.apache.org Subject: Re: Kafka Connect Rest Extension Question Hello Yang Hyung Wook, In your post I do not see anything obviously wrong, so you may need to do some more debugging. 1. Are

Re: Kafka Connect Rest Extension Question

Hello Yang Hyung Wook, In your post I do not see anything obviously wrong, so you may need to do some more debugging. 1. Are you using the same jar for both the classpath and plugin.path tests? If not, do they both contain the service loader manifest file? You can test this with https://docs.orac

Kafka Connect Rest Extension Question

https://stackoverflow.com/questions/76797743/how-can-i-solve-connectrestextension-error There is an issue with this link where the ConnectRestExtension implementation is not registered. I've done everything the kip official documentation says, but can you tell me why it doesn't work? 양형욱 Yang

Re: MirrorMaker 2.0 question

I got it to work by raising a kafka connect from which to launch the mm2. Silly question. El lun, 20 mar 2023 a las 23:09, Miguel Ángel Fernández Fernández (< miguelangelprogramac...@gmail.com>) escribió: > I have two clusters up on the same machine with docker-compose >

Re: MirrorMaker 2.0 question

I have two clusters up on the same machine with docker-compose services: zookeeper-lab: image: "bitnami/zookeeper:3.8.1" restart: always environment: ZOO_PORT_NUMBER: 2183 ALLOW_ANONYMOUS_LOGIN: "yes" ports: - "2183:2183" - "2886:2888" - "3886:3888" kafk

Re: MirrorMaker 2.0 question

Mi Miguel, How many nodes are you running MM2 with? Just one? Separately, do you notice anything at ERROR level in the logs? Cheers, Chris On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández < miguelangelprogramac...@gmail.com> wrote: > Hello, > > I'm doing some tests with Mirror

MirrorMaker 2.0 question

Hello, I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of kafka clusters, I think everything is set up correctly. However, when I run /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties the result I get is the creation of the topics mm2-configs.A.internal, mm2-of

The question about start kafka-server

Dear developer:       Hello,get to the point,when I start kafka-server(kafka_2.13-3.1.0), I have encountered a prolem, I have put the relational files on attachment position, please help me to check the problems and give some solutions, I will be very happy for your help. Looking forward to

Re: Question about KRaft

> > On Fri, Mar 10, 2023 at 12:54 PM sunil chaudhari < > sunilmchaudhar...@gmail.com> wrote: > > > Hi Luke, > > This docu is good. > > Does it apply for confluent as well? > > > > > > > > On Fri, 10 Mar 2023 at 8:47 AM, Luke Chen

Re: Question about KRaft

0 Mar 2023 at 8:47 AM, Luke Chen wrote: > > > Hi Zhenyu, > > > > Answering your question: > > > > > Should I simply > > 1. download 3.4 binary > > 2. stop ZK & Kafka service > > 3. upgrade Kafka to 3.4 > > 4. start only Kafka service wit

Re: Question about KRaft

Hi Luke, This docu is good. Does it apply for confluent as well? On Fri, 10 Mar 2023 at 8:47 AM, Luke Chen wrote: > Hi Zhenyu, > > Answering your question: > > > Should I simply > 1. download 3.4 binary > 2. stop ZK & Kafka service > 3. upgrade Kafka to 3.4 >

Re: Question about KRaft

Hi Zhenyu, Answering your question: > Should I simply 1. download 3.4 binary 2. stop ZK & Kafka service 3. upgrade Kafka to 3.4 4. start only Kafka service with KRaft server.properties That is not migrating, actually. That is just creating another kafka cluster in KRaft mode. The po

Re: Question about REQUEST QUEUE size

Thanks for your reply, after that I figured out my total producer rate is just around 8000/s although message rate is around 80K - 100K (message in per sec metric). Is 8000 requests per sec are simply too tiny to have any significant changes in request queue size ? May I ask you some additional qu

Re: Question about KRaft

Hi Sunil, As mentioned earlier in my question, I have only one "combined" node as both controller and broker, and I totally accept downtime (stop service) So just want to ask for my case, single node, if I want to upgrade to 3.4 then start service under KRaft (get rid of ZK), what wo

Kafka General Question

Hello Kafka Team I am trying to some POC of the Messaging and it will working fine with the remote host and client . But I have question how I can send message/response from consumer to producer or server using topic . Please Guide me on this and share any doc if possible . Thank You Rohan Kar

Re: Question about REQUEST QUEUE size

The metric that you are monitoring is correct and works correctly in my setup for the versions that you mentioned. 1. To rule out any problems with reading via jconsole, do you see a non-zero value for "kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=([-.\w]+)" ? 2. Do y

Question about REQUEST QUEUE size

Hi guys, According to the document we can get the request queue size (the one network threads enqueue, IO threads dequeue and will block network thread if full) via "kafka.network:type=RequestChannel,name=RequestQueueSize". However, when I use jconsole to view this metric it always = 0. I have con

Re: Question about KRaft

How will you achieve zero downtime of you stop zookeeper and kafka? There must be some standard steps so that stop zookeeper one by one and start kraft same time so that it will be migrated gradually. On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang wrote: > Hi team, > > Here is a quest

Question about KRaft

Hi team, Here is a question about KRaft from normal user, who starts to use and learn Kafka since 3.2 Last month Kafka 3.4, the first bridge release was available, and I am considering to have a plan to use KRaft (get rid of ZK) since this version Currently I am using 3.3.2 (upgrade from 3.2

Re: Question regarding Kafka compaction and tombstone

ednesday, February 15, 2023 9:31 PM To: users@kafka.apache.org Subject: Question regarding Kafka compaction and tombstone Hi, I am using Kafka 3.2 (in windows) and for a topic i to send tombstone records. Everything was ok but i always see last value for the key (even i see null

Question regarding Kafka compaction and tombstone

Hi, I am using Kafka 3.2 (in windows) and for a topic i to send tombstone records. Everything was ok but i always see last value for the key (even i see null records present after delete.retention.ms period) Example Key1 value1 Key2 value2 Key1 - null record - tombstone record and so on I am

Re: My Question About Network Partition

Not necessarily, depends on how your producer is configured. As an example, if your producer were configured with acks=all and min.isr = 2 (which is a pretty typical config), then you wouldn't be able to produce messages since the brokers would be unable to replicate the data to other broker(s) in

Re: Some question about the kafka website language

n use English, not bad. > > > > > -- 原始邮件 -- > 发件人: > "users" > < > xianshengzim...@gmail.com>; > 发送时间: 20

Re: Some question about the kafka website language

To sunil chaudhari:Hi, You are right, but the configure is english that are unimporment. To Daniel Hinojosa：Hi, I'm from China. I'm interested in Chinese. - About "Translate it" : It is a good way for "Translate to", but it also has some porblem.If my English is not good, I have frequently

Re: Some question about the kafka website language

I am just wondering even if you read document in your language, but ultimately server.config and server.log will be in english only.. right? On Tue, 31 Jan 2023 at 9:00 PM, Nengda Ouyang wrote: > I see the kafka website only have language version. Why doesn't provide > other language. >

Re: Some question about the kafka website language

I can only assume that no one has come forward to translate. You can in most browsers, translate the page. In Chrome, you can right-click on the page and select "Translate to" and change to whatever language you need. It may not be perfect but it would get you going. Here is a snapshot (Not sure i

Re: Some question about the kafka website language

I mean our have English website .Why doesn't provide other language. >

Some question about the kafka website language

I see the kafka website only have language version. Why doesn't provide other language.

Re: My Question About Network Partition

Yes you can. data replication is in async mode. On Mon, Jan 30, 2023 at 12:27 AM 大雁小鱼 <747121...@qq.com.invalid> wrote: > Can producers send messages if all brokers in a Kafka cluster are unable > to communicate with each other while the connection between brokers and > Zookeeper is normal? Thank

About a little white question

Hello! Recently, I have an error in the process of using logstash to consume kafka and send it to bigquery ``` [2022-12-13T17:06:33,453][ERROR][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][main][1f6dc206229592c94cf83964f9811eb491c35bc7029e9096c5e53f3d890cc9be] [Consumer clien

Re: question: kafka stream Tumbling Window can't close when no producer sending message

Hello, Yes, this is correct. There is a difference between what we call “stream time” and regular “wall-clock time”. All the windowing operations need to be deterministic, otherwise your results would depend on when you run your program. For that reason, we have “stream time”, which takes its

question: kafka stream Tumbling Window can't close when no producer sending message

Dear kafka mangers: when I use kafka stream Tumbling Window, i found that if the producer send messages to source topic continuously, the stream TimeWindow could work as expected. but once i stop the producer, the last window couldn't close, until the producer send next message. Is this n

Question about log lock removal

Hi everybody, we’ve been seeing problems on restart of our kafka brokers, trying to acquire the lock before shutdown. After about 45 minutes, the restart is possible but that makes the deployment too long. We are considering adding a preStep helm job to remove the lock before restarting, but we’

Question about MM2's source-cluster.checkpoints.internal topic

Hi, I am using MM2 version 3.1.0 for a uni-directional mirroring of source-cluster -> target-cluster. I am confused with the values found in the source-cluster.checkpoints.internal topic at the target cluster when the source's earliest offset is not 0. In my test setup, I prepared a topic samples.

Re: Kafka Question

reaming events. However, you can easily write a > program that consumes those events from Kafka and then does what you want > :) > > Cheers, > > Liam > > On Tue, 3 May 2022 at 06:30, Emily Schepisi > wrote: > > > Hello, > > > > I have a question about

Re: Kafka Question

Emily Schepisi wrote: > Hello, > > I have a question about Kafka. If I put an upper and lower control limit on > the data, and the log records an event where the upper or lower control > limit is breached, will Kafka be able to send a notification via email or > text message to th

Kafka Question

Hello, I have a question about Kafka. If I put an upper and lower control limit on the data, and the log records an event where the upper or lower control limit is breached, will Kafka be able to send a notification via email or text message to the user? Example: I'm tracking the

Re: [Question] What is the best practice in consuming multiple topics?

Hi BaiYang, Thanks for your reply! For your question, my answer is below: I don’t need to care about Kafka brokers because I use AWS’s MSK product. For producers, I didn’t restart it, which is a cluster services and provided by other team.Consumers will be restarted when my application is

Re: [Question] What is the best practice in consuming multiple topics?

how about stop the kafka brokers and restart them and restart producers and consumers? 回复的原邮件 | 发件人 | Xiong, Pin | | 日期 | 2022年03月31日 22:14 | | 收件人 | users@kafka.apache.org | | 抄送至 | | | 主题 | [Question] What is the best practice in consuming multiple topics? | To whom it may concern

[Question] What is the best practice in consuming multiple topics?

To whom it may concern, I have a very different problem when using Kafka, and hope someone can help me! Background I need to consume several different topics in my java project. For each topic, they has different configurations, such as max.pull.records, but the same consumer group. Also, the

Question about RecordBatch truncation in the kafka protocol

Hi all, I'm working on a client library for kafka, and I can't seem to find the difference between truncation of a RecordBatch due to log compaction and truncation of a RecordBatch due to the MaxBytes requested by the reader, is there a flag set in the message that I can reference? https://cwiki.ap

Config Provider Question

Hi All, Can the Config Provider be specified at the connector level or is the configuration only available at the worker level (docker configuration)? Here are our observations: 1. When we specify the config provider at the connector level then when adding the connector, the validation for

Re: Question about kafka log retention

Hello Xiaochi , I am not sure if I have understood the problem correctly but beware the fact that only old log segments and not the current log segment are taken into account for deletions. So if you want the data to be deleted in a timely manner, you also need to configure a tighter interval

Question about kafka log retention

Hello, I am currently using kafka 3.1.0 with java 1.8. I have set kafka log retention policy in the server.properties like this: log.retention.hours=6 log.retention.bytes=5368709120 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 log.cleanup.policy=delete However, it see

question: kafka connect mm2 curl command

I am running AWS MSK v2.7.0 with SASL/SCRAM. I have an ec2 instance running the kafka connector and mirror maker 2 (MM2). I was able to start up the distributor w/o any issue but when I run the curl command to start the worker, the output look fine but no data are replicating. curl -s -X POST -H '

Re: Kafka Streams Testing Suite Question

noticed is that for the TopologyTestDriver this is never called, causing some issues for our stream tests. I also confirmed that for the actual streams code this method is always called when starting the stream instances. So my question is that is this case (and possibly others I’m not aware of

Kafka Streams Testing Suite Question

, causing some issues for our stream tests. I also confirmed that for the actual streams code this method is always called when starting the stream instances. So my question is that is this case (and possibly others I’m not aware of) valid for updating the testing suite drivers to more closely

Re: question about mm2 on consumer group offset mirroring

es consumer group offsets. > > Ryanne > >> On Thu, Sep 30, 2021, 9:18 AM Calvin Chen wrote: >> >> Hi all >> >> I have a question about the mirror make 2, on the consumer group offset >> mirroring, what is the duration for mm2 to detect consumer group offset &g

Re: question about mm2 on consumer group offset mirroring

Hey Calvin, the property you're looking for is emit.checkpoint.interval.seconds. That's how often MM will write checkpoints, which includes consumer group offsets. Ryanne On Thu, Sep 30, 2021, 9:18 AM Calvin Chen wrote: > Hi all > > I have a question about the mirror make

question about mm2 on consumer group

Hi all I see for mm2, mirrored topic name will have source kafka cluster put as prefix, but consumer group doesn't, why not put consumer group naming same as topic? is it because consumer group/offset mirroring not only from source-kafka to remote-kafka but also from remote-kafka to source-kafk

question about mm2 on consumer group offset mirroring

Hi all I have a question about the mirror make 2, on the consumer group offset mirroring, what is the duration for mm2 to detect consumer group offset change and mirror it to remote kafka consumer group? I have my mm2 code define as below: {{ kafka01_name }}->{{ kafka02_n

Re: Question about controller behavior when fenced

allocation roles/duties of a controller. The old controller will not shutdown. It will become one of the non-controller brokers, if I may call it. Regards, Neeraj On Thursday, 30 September, 2021, 12:12:08 am GMT+10, Andrew Grant wrote: Hi all, I had a question about controller

Question about controller behavior when fenced

Hi all, I had a question about controller behavior when fenced. From my understanding epoch numbers are used to fence brokers who might think they're still the controller but really in the meantime a new broker has been elected as the new controller. My question is, how does a broker realize

Re: MirrorMaker 2.0 compliance question

yep! On Wed, Jul 21, 2021, 3:18 AM Tomer Zeltzer wrote: > Hi, > > Can I use MirrorMaker2.0 from Kafka 2.8.0 with Kafka version 2.4.0? > > Thanks, > Tomer Zeltzer > > This email and the information contained herein is proprietary and > confidential and subject to the Amdocs Email Terms of Ser

MirrorMaker 2.0 compliance question

Hi, Can I use MirrorMaker2.0 from Kafka 2.8.0 with Kafka version 2.4.0? Thanks, Tomer Zeltzer This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-o

Re: Question about Kafka ACL

ge: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Kafka Summit] <https://www.kafka-summit.org/> On Wed, Jun 9, 2021 at 9:42 PM Calvin Chen wrote: > Hi all > > I have a question about kafka acl, is it possible to limit user

Question about Kafka ACL

Hi all I have a question about kafka acl, is it possible to limit users to access a topic? For example, if person-a created kafka-user-a and grant kafka-user-a to access topic-a, and person-b knows there is topic-a, but he doesn't know the credential to access topic-a via kafka-user-a,

Re: Yet Another Repartitioning Question About Kafka Streams

follow a cleanup delay policy. Guozhang On Sun, Apr 11, 2021 at 11:13 AM Gareth Collins wrote: > Hi Guozheng, > > Thanks very much again for the answers! > > One follow-up on the first question. Just so I understand it, how would it > know where to continue from? > I woul

Re: Yet Another Repartitioning Question About Kafka Streams

Hi Guozheng, Thanks very much again for the answers! One follow-up on the first question. Just so I understand it, how would it know where to continue from? I would assume that once we repartition, the new node will own the position in the consumer group for the relevant partition(s) so Kafka

Re: Yet Another Repartitioning Question About Kafka Streams

Hello Gareth, 1) For this scenario, its state should be reusable and we do not need to read from scratch from Kafka to rebuild. 2) "Warmup replicas" is just a special standby replica that is temporary, note that if there's no partition migration needed at the moment, the num.warmup.replicas is ac

Yet Another Repartitioning Question About Kafka Streams

Hi, Thanks very much for answers to my previous questions here. I had a couple more questions about repartitioning and I just want to confirm my understanding. (1) Given the following scenario: (a) I have a cluster of Kafka stream nodes with partitions assigned to each. (b) One node goes down.

Re: automatic offset reset question

Hello, Can you check the offsets.retention.minutes paramether >From the documentation ( https://kafka.apache.org/documentation/#upgrade_200_notable) : - KIP-186 increases the default offset retention time from 1 day to 7 days. This makes it l

automatic offset reset question

Hi, I have a stream process that from time to time starts re-ingesting old data which affects the whole pipeline. What I think is happening is the consumer group gets reset to the earliest, but since: * the consumer group name is always the same * Replica set is 3 * auto.offset.reset is 'none

A question related to quota in Kafka

Hello everyone. I send you this mail because I have a question related to quota in Kafka. I read the following articles there : - https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas - https://kafka.apache.org/documentation/#design_quotas My Kafka cluster has the version

  1   2   3   4   5   6   7   8   9   10   >