Re: How to alter topic config with broker endpoint

Hi Weichu, Use --bootstrap-server option instead. ex: *bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name test --alter --add-config max.message.bytes=124000* Thanks. On Tue, Apr 14, 2020 at 12:17 PM Weichu Liu wrote: > Hi, > > We have an in-house Kafka cl

Re: Is that a bug?

Yes, for more specific, it's the property: *allow.auto.create.topics, *which default is set to true. On Fri, May 22, 2020 at 5:07 PM M. Manna wrote: > Hey Xie, > > On Fri, 22 May 2020 at 08:31, Jiamei Xie wrote: > > > Hi > > > > Kill all zookeeper and kafka process. Clear zookeeper and kafka da

Re: Request for adding to contributors list

Hi Guruprasad, What Matthias was asking, is what your JIRA account username is? So that he or other committer can help you grant you the permission to assign JIRA tickets. If you haven't got the JIRA account yet, you can sign up here: https://issues.apache.org/jira/secure/Signup!default.jspa Thank

Re: Re: subscribe comfirm

Hi Roc, The 'signature' is not related to the subscription. Again, if you want to subscribe to the mailing list, please follow the instructions from the web page: https://kafka.apache.org/contact. That is, to send an email to users-subscr...@kafka.apache.org dire

Re: New Website Layout

When entering streams doc, it'll always show: *You're viewing documentation for an older version of Kafka - check out our current documentation here.* On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford wrote: > Thanks for the PR and feedback Michael. Appreciated. > > On Wed, 5 Aug 2020 at 10:49, Mick

Re: New Website Layout

the new layout is the tags are > > > empty, > > > > and no longer enclose the config name, > > > > > > > > > > > > > > > > batch.size > > > > > > > > > > > > meaning you can

Re: Should kafka be deleting unused replicas?

Hi Dima, They'll stick around. I think the point is that when node 3 is back, you should do partition reassignment so that the pre-existed partition/replica data can be reused. But for your question, the unused log won't delete is a bug or as designed... I think it is working as designed and it sh

Re: Kafka TImeout & Network Exception (Urgent)

Maybe you can reduce the *batch.size* and *linger.ms * value and see if that issue still happens. Also check the *acks* value since it'll also affect the response time before collecting enough acks from nodes. On Wed, Sep 16, 2020 at 10:56 AM Navneeth Krishnan wrote: > Hi Shaoh

Re: [ANNOUNCE] New committer: Chia-Ping Tsai

Congratulations! Chia-Ping大大! Well deserved! Luke On Tue, Oct 20, 2020 at 2:30 AM Mickael Maison wrote: > Congrats Chia-Ping! > > On Mon, Oct 19, 2020 at 8:29 PM Ismael Juma wrote: > > > > Congratulations Chia-Ping! > > > > Ismael > > > > On Mon, Oct 19, 2020 at 10:25 AM Guozhang Wang > wrote

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

Congratulations, Sophie! You always provide good review comments for my PRs. Well deserved! Luke On Tue, Oct 20, 2020 at 9:23 AM Rankesh Kumar wrote: > Many congratulations, Sophie. > > Best regards, > Rankesh > > > > On 20-Oct-2020, at 12:34 AM, Gwen Shapira wrote: > > > > Congratulations, So

Re: Subscribe to Kafka dev mailing list

Hi, You should send an email to dev-subscr...@kafka.apache.org to subscribe to Kafka dev mailing list. Luke Thanks. On Thu, Dec 3, 2020 at 12:40 AM 阿满 <6065...@qq.com> wrote: > Subscribe to Kafka dev mailing list

Re: Kafka transactions commit message consumability issues

Hi, What Boyang meant, is this consumer config: *isolation.level, *which default is *read_uncommitted*, so the consumer will be able to read all messages with or without committed. ref: https://kafka.apache.org/documentation/#consumerconfigs_isolation.level Thanks Luke On Wed, Jan 27, 2021 at 2:

Re: options for kafka cluster backup?

Hi Pushkar, MirrorMaker is what you're looking for. ref: https://kafka.apache.org/documentation/#georeplication-mirrormaker Thanks. Luke On Fri, Mar 5, 2021 at 1:50 PM Pushkar Deole wrote: > Hi All, > > I was looking for some options to backup a running kafka cluster, for > disaster recovery re

Re: Sticky Assignor logic : prioritrize consumers with less number of partitions when reassigning.

Hi Mazen, I'm not sure if I understand your question correctly. For your question: > 3 consumers 10 partitions and the reason for rebelancing is the > addition of a new consumer C4 say C1 has 3 partitions, C2 has 5 > partitions and C3 has 3 partitions the case where C4 shall handle m

Re: Sticky Assignor logic : prioritrize consumers with less number of partitions when reassigning.

IP-54+-+Sticky+Partition+Assignment+Strategy Please let me know if you have other questions. Thanks. Luke On Tue, Mar 9, 2021 at 2:52 PM Luke Chen wrote: > Hi Mazen, > I'm not sure if I understand your question correctly. > For your question: > >> 3 consumers 10 partit

Re: [VOTE] 2.6.2 RC0

Hi Sophie, A small doc update for 2.6.2. I think we missed it. https://github.com/apache/kafka/pull/10328 Thanks. Luke On Tue, Mar 16, 2021 at 11:12 AM Guozhang Wang wrote: > Hi Sophie, > > I've reviewed the javadocs / release notes / documentations, and they LGTM. > > +1. > > > Guozhang > > On

Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck

Congratulations Bill! Luke On Thu, Apr 8, 2021 at 9:17 AM Matthias J. Sax wrote: > Hi, > > It's my pleasure to announce that Bill Bejeck in now a member of the > Kafka PMC. > > Bill has been a Kafka committer since Feb 2019. He has remained > active in the community since becoming a committer.

Re: [ANNOUNCE] New Committer: Bruno Cadonna

Congrats Bruno!! Luke On Thu, Apr 8, 2021 at 9:18 AM Matthias J. Sax wrote: > Congrats Bruno! Very well deserved! > > > -Matthias > > On 4/7/21 3:51 PM, Bill Bejeck wrote: > > Congrats Bruno! Well deserved. > > > > Bill > > > > On Wed, Apr 7, 2021 at 6:34 PM Guozhang Wang wrote: > > > >> Hello

Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch

Congratulations Randall! Luke Bill Bejeck 於 2021年4月17日 週六 上午11:33 寫道： > Congratulations Randall! > > -Bill > > On Fri, Apr 16, 2021 at 11:10 PM lobo xu wrote: > > > Congrats Randall > > >

Re: Kafka : Version Upgrade

Hi, About the upgrade documentation, please read here: https://kafka.apache.org/documentation/#upgrade It should answer your 2 questions. Thanks. Luke On Tue, Jun 8, 2021 at 6:01 PM Kafka Life wrote: > Dear Kafka Experts > > 1- Can any one share the upgrade plan with steps /Plan /tracker or an

Re: Kafka incremental sticky rebalancing.

Hi Mazen, Did you auto commit offsets or manually? When do you commit offsets? Thanks Luke Mazen Ezzeddine 於 2021年6月21日 週一 下午7:42 寫道： > I am running Kafka on Kubernetes using the Kafka Strimzi operator. I am > using incremental sticky rebalance strategy by configuring my consumers > with the fo

Re: Kafka incremental sticky rebalancing.

< mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > Hi Luke, > > I am committing the offsets manually, and after processing each of the > records returned by one poll loop iteration using consumer.commitSync(). > > Regards, > > ____ > From: L

Re: Consumer Group Stuck on "Completing-Rebalance" State

Hi Tao, 1. So this bug only applies to cooperative-sticky assignor? --> Yes, this bug only applies to sticky assignor (both eager and cooperative) since we will refer to the consumer's previous assignment. 2. Does assignor strategy (cooperative-sticky vs sticky vs others) really matter in this cas

Re: Consumer Group Stuck on "Completing-Rebalance" State

ssue. > > On Wed, Jun 23, 2021 at 4:38 PM Luke Chen wrote: > > > Hi Tao, > > 1. So this bug only applies to cooperative-sticky assignor? > > --> Yes, this bug only applies to sticky assignor (both eager and > > cooperative) since we will refer to the consumer&#x

Re: Consumer Group Stuck on "Completing-Rebalance" State

of partitions it works on. For such pattern, > I think I may use Round-Robin strategy. Any suggestions? > > Thanks! > > On 2021/06/24 00:15:57, Luke Chen wrote: > > Hi Sophie, > > Thanks for your clarification. :) > > > > Luke > > > > Sophie Blee

Re: kafka partition_assignment_strategy

Hi Marcus, What Ran meant is that please run the following command to check the result. bin/kafka-topics.sh --describe --topic TOPIC_NAME --bootstrap-server localhost:9092 Thanks. Luke On Thu, Jul 1, 2021 at 8:06 PM Ran Lupovich wrote: > I meant kafka-topics describe command , I want to make

Re: consumer group exit :explanation

Hi, In addition to the above suggestion, I think you can also check the broker logs clearly. You should see the reason. For example, here's what I saw in my logs: [2021-02-26 16:38:45,777] INFO [GroupCoordinator 0]: Member consumer-groupId-1-7e24136f-8d7e-4303-a150-32dae06c18c0 in group groupId ha

Re: Kafka incremental sticky rebalancing.

additional delays in addition to rebalancing delays > (the failed one due to the exception raised, and the one caused by > restarting). I don’t think this is normal? > > Anyway, could you please give me a hint on why this is happening, can it > be because the rebalancing is taking pl

Re: Querying a consumer group during a rebalance using admin API

Hi Mazen, We'll check if the exception returned from brokers is retriable exceptions, if yes, we'll keep retrying until request timeout(default 30 seconds). Some retriable exceptions like: NOT_COORDINATOR, COORDINATOR_NOT_AVAILABLE, LEADER_NOT_AVAILABLE,... Thanks. Luke On Thu, Aug 5, 2021 at 4:1

Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD

Hi Yoda, For your question: > If an application gets an uncaught exception, then the failed thread will be replaced with another thread and it will continue processing messages, skipping that failed message? --> Yes, if everything goes well after `replace thread`, you can ignore this failed messag

Re: Kafka attempt rediscovery never happen

Hi Antonio, What you can do is to check the broker side log, to see if there's any info. If no, you can also enable DEBUG log to have more info for troubleshooting. Thank you. Luke On Mon, Aug 23, 2021 at 11:27 PM Antonio Pires wrote: > Hello fellow Kafka users, > > I've been trying to understa

Re: Security vulnerabilities in kafka:2.13-2.6.0/2.7.0 docker image

Hi Ashish, I suggested that you upgrade to V2.8. I checked 2 of the CVEs, and are fixed (or not used, like libfetch) in V2.8. If you still found the CVEs existed in V2.8, please raise it. Thank you. Luke On Wed, Sep 1, 2021 at 4:07 AM Ashish Patil wrote: > Hi Team > > I wanted to use the 2.6

Re: [EXTERNAL] Re: Security vulnerabilities in kafka:2.13-2.6.0/2.7.0 docker image

m:* Jake Murphy Smith > *Sent:* 01 September 2021 09:31 > *To:* Ashish Patil > *Subject:* RE: [EXTERNAL] Re: Security vulnerabilities in > kafka:2.13-2.6.0/2.7.0 docker image > > > > > > > > *From:* Luke Chen > *Sent:* 01 September 2021 04:11 > *To:* Kafka U

Re: kafka producer hangs

Hi Владислав, How do you know "they do not reach the kafka server."? We need to know your configs on producer and server side so that we can help you. Also, you might be able to turn on TRACE or DEBUG log on producer and server for troubleshooting. If possible, a complete log would be better than

Re: How is __consumer_offsets partitioned?

Hi Michał, Internally, Kafka uses "*consumer group ID*" as the key to decide which __consumer_offsets partition to be used. The code is like this: `Utils.abs(groupId.hashCode) % groupMetadataTopicPartitionCount` You can check the `partitionFor` method in GroupMetadataManager class. Hope that hel

Re: Kafka __consumer_offset topic

Hi David, Glad that my old post helps you! :) Thank you. Luke On Sat, Sep 11, 2021 at 6:39 AM David Ballano Fernandez < dfernan...@demonware.net> wrote: > oops, I think I found my answer in a few mails below: > and if i am not mistaken that is why my only consumer is only writing to > one partit

Re: Reading from lagging replicas

Hi Michał, I think you want to check the term in Kafka: In Sync Replicas (a.k.a ISR). You might want to check the "Replication" section in doc here . The configuration is this one: *replica.lag.time.max.ms * https

Re: does kafka provide REST API for producing events to topic?

Hi Pushkar, > I would like to know if kafka also provides any REST API which can be accessed by the test client (rather than making a TCP connection to broker) and then invoking those REST API to produce events onto topics? --> Sorry, Kafka only provides TCP connection to brokers. Thank you. Luke

Re: [kafka-clients] [VOTE] 2.7.2 RC0

Hi Mickael, Thanks for the release. I did: 1. Verified checksums and signatures 2. Run quick start steps 3. Verified the CVE-2021-38153 is indeed fixed in kafka-2.7.2-src.tgz . +1 (non-binding) Thank you. Luke On Tue, Oct 26

Re: New Kafka Consumer : unknown member id

Hi, Which version of kafka client are you using? I can't find this error message in the source code. When googling this error message, it showed the error is in Kafka v0.9. Could you try to use the V3.0.0 and see if that issue still exist? Thank you. Luke On Thu, Oct 28, 2021 at 11:15 PM Kafka L

Re: Stream to KTable internals

Hi Chad, > I'm wondering if someone can point me to the Kafka streams internal code that reads records for the join? --> You can check StreamThread#pollPhase, where stream thread (main consumer) periodically poll records. And then, it'll process each topology node with these polled records in strea

Re: Kafka Subscription

Hi Sabari, To subscribe, you should send an email to users-subscr...@kafka.apache.org. Not users@kafka.apache.org. Thanks. Luke On Sun, Oct 31, 2021 at 11:46 PM Sabari MayaKrishnan wrote: > Hi, > > For Kafka subscription >

Re: Producer Timeout issue in kafka streams task

Hi Pushkar, In addition to Matthias and Guozhang's answer and clear explanation, I think there's still one thing you should focus on: > I could see that 2 of the 3 brokers restarted at the same time. It's a total 3 brokers cluster, and suddenly, 2 of them are broken. You should try to find out th

Re: Log cleaner is marking the partition as uncleanable

Hi Dhirendra, The error message said that the log cleaner believed your log file is corrupted because there's a record with 0 Byte, but the minimum record size is 14 Bytes (including CRC, Key, Value,...). I don't think it's good to fix the corrupted file manually. Thank you. Luke On Mon, Nov 15,

Re: EOL clarification

Hi Piyush, Usually, Kafka only produces releases for the 2 most recent branches. Ex: We're going to release V3.1, and some fix will also back port to V3.0 branch. So, from the download page, you can see most minor version is only up to 1, ex: 2.3.1, 2.5.1... But that's not a strict rule, so you'll

Re: EOL clarification

am on 2.4.1 right now. If it's receiving security and > bug fixes then I don't see any need to upgrade to latest version 3.0 or > even 2.8.1 > > Thanks and Regards > Piyush Mittal > > > On Tue, Nov 16, 2021 at 12:38 PM Luke Chen wrote: > > > Hi Piyush, >

Re: Re: Accessing TLS certs and keys from Vault into Kafka

Hello, > can you provide any existing implementations for reference? You can check the `FileConfigProvider` in clients/src/main/java/org/apache/kafka/common/config/provider/FileConfigProvider.java. Or check the repo: https://github.com/jcustenborder/kafka-config-provider-vault Hope this helps. T

Re: Setting Memory Bound RocksDB Config

Hi Gray, Answering your question below: > If we set TOTAL_OFF_HEAP_MEMORY, to, say, 1GB, does that mean the app should only use 1GB for the whole application, or is it per store (so 3GB total)? Or perhaps per store and per task (3GB * num tasks)? The TOTAL_OFF_HEAP_MEMORY setting limits the memor

Re: Changing Partitions of kafka

Hello Rajat, I'm not sure what you mean to "reshuffle messages", because once the messages are stored in brokers, they can't be modified anymore. But if you want to make the previous added messages route to the same partitions after partition increasing, you can write custom partitioner: https://k

Re: uneven distribution of events across kafka topic partitions for small number of unique keys

Hello Pushkar, Default distribution algorithm is by "hash(key) % partition_count", so there's possibility to have the uneven distribution you saw. Yes, there's a way to solve your problem: custom partitioner: https://kafka.apache.org/documentation/#producerconfigs_partitioner.class You can check

Re: Changing Partitions of kafka

s > first before I start accepting the new data. Right? > A separate partitioner code has to be executed post increase in partitions > which will read all records in kafka topic and run partition algorithm and > then push it to new partition number simply. , then we can start accepting >

Re: Consumer failure after rolling Broker upgrade

Hi James, > Bouncing the clients resolved the issue Could you please describe which version you upgrade to, to resolve this issue? That should also help other users encountering the same issue. And the code snippet you listed, existed since 2018, I don't think there is any problem there. Maybe the

Re: delete the topic, and immediately create the topic

Hi 志恒, > When I use AdminClient to delete the topic, and immediately create the topic that was just deleted, it always indicates that the topic already exists In Kafka, the topics info (metadata) is stored in cache, and will not be updated immediately. That's why you saw this situation. I don't thi

Re: EOL clarification

Is that correct? > > Thanks and Regards > Piyush Mittal > > > On Tue, Nov 16, 2021 at 1:41 PM Luke Chen wrote: > > > Hi Piyush, > > > As an example, let's say I am on 2.4.1 right now. If it's receiving > > security and bug fixes then I don't see

Re: Kafka mTLS authentication

Hi Yingjie, > However, I meet a problem. If I need to add, remove or renew the certificate to Kafka’s truststore, Kafka requires a reboot which would impact the service available for other teams. > So I want to know if there is a better way to support the change of Kafka’s certificate without im

Re: Kafka mTLS authentication

Hi Yingjie, No worries! Glad to help! Luke On Tue, Nov 23, 2021 at 5:52 PM yingjie zou wrote: > Hi Luke, > > This solved my problem. > I'm sorry to trouble you because I didn't read the document carefully. > Thank you very much. > > > Yingjie Zou > > O

Re: TLS certificate error with PEM AND KAFKA

Hi Sai Chandra Mouli T, Glad you found out the PKCS#8 pbe-sha1-rc4-128 (-v1) works well. In Kafka's unit test, we use the algorithm: pbeWithSHA1And3-KeyTripleDES-CBC (v1) to encrypt the key to do test. Actually, the v1/v2 means the algorithm in PCKS#5 v1/v2 (check here: https://www.openssl.org/d

Re: Kafka Streams - left join behavior

Hi Miguel, > Is there a way to force the behavior I need, meaning... using left join and a JoinWindows output only one message (A,B) or (A, null) I think you can try to achieve it by using *KStream-GlobalKTable left join*, where the GlobalKTable should read all records at the right topic, and then

Re: How to configure Spring Boot Kafka client so it does not try to connect

Hi Honza, > The problem is that the unsuccessful attempts to connect take about 30-40 seconds and our tests are significantly slowed down. I think the reason why it retried for 30-40 seconds is that you tried to create topics via adminClient (new KafkaAdmin.NewTopics) as described in the Spring-Kaf

Re: Rebalancing taking more than 15/20 minutes to complete

Hi Michael, I'm not 100% sure if you've hit the issue on KAFKA-9752 . >From the description in the KAFKA-9752 , you are: 1) we have a consumer using an old JoinGroup version, 2) the consumer times ou

Re: Event order in Kafka Streams after Left Join

Hi Miguel, Of course you can use "Processor API" to achieve what you want. But it needs more coding. Alternatively, I think you can define a better value for "*max.task.idle.ms *" configuration. Default value is 0, which means it basically doesn't wait for more data in emp

Re: SASL_PLAIN configuration problems

Hello Andrzej, The error: *NetworkClient - [Producer clientId=signings] Connection to node -1 (localhost/127.0.0.1:9093 ) failed authentication due to: Unexpected handshake request with client mechanism PLAIN, enabled mechanisms are []* It is complaining that your client is

Re: Use case: Per tenant deployments talking to multi tenant kafka cluster

Hi Christian, Answering your question below: > Let's assume we just have one topic with 10 partitions for simplicity. We can now use the environment id as a key for the messages to make sure the messages of each environment arrive in order while sharing the load on the partitions. > Now we want e

Re: AccessDeniedException in Kafka on Windows

Hi Martijn de Bruijn, I think I've reponsed your other mail. But you're right: > On several places FileChannel.open(path, StandardOpenOption.READ) is called which will always fail on Windows when path is a directory. Yes, we tried to flush(fsync) data on directory to make sure it's successfully

Re: Use case: Per tenant deployments talking to multi tenant kafka cluster

o the topic like > described above? > > Christian > > Am Mi., 8. Dez. 2021 um 11:19 Uhr schrieb Luke Chen : > > > Hi Christian, > > Answering your question below: > > > > > Let's assume we just have one topic with 10 partitions for simplicity. >

Re: CVE-2021-44228 – Log4j 2 Vulnerability

Hi David Ballano Fernandez, Thanks for reporting this issue. Yes, this is the most critical 0-day vulnerability for security members. I've been investigating this CVE for a while, and I confirmed that* log4j 1.x versions are not affected by this vulnerability.* That is, *Kafka, which is using log4

Re: Issue on Partition Assignment

Hi Tao, I think we need the logs (client and broker side) to know clearly what happened there. Besides, I can't see the image you attached under "Consumer Group State (Please note that app008 is kicked out)" section. One thing to note is that, we've fixed some bugs related to rebalance and partit

Re: consumer group and current offsets

Hi NicolaeMarasoiu, It looks like a race condition during consumer group deletion and new group creation. Some questions: > However, in a particular case, I deleted the consumer group, and yet, a program creates a consumer group which has the current offset of the deleted group... How do you k

Re: CVE-2021-44228 – Log4j 2 Vulnerability

input.* So, in short, as long as you're using Kafka, and not setting the jms configuration: *TopicBindingName* or *TopicConnectionFactoryBindingName *to something that JNDI can handle, it is safe! Thank you. Luke On Sat, Dec 11, 2021 at 4:23 PM Luke Chen wrote: > Hi David Ballano Fernand

Re: Re: Issue on Partition Assignment

partition 0 from app08, since rebalance event > above ] > 2021-12-09 04:39:39 Received message from topic > ***InternalLoggingMetadataInfo, partition: 0, offset: 375, RequestId: > b9740fdd-bc1c-4ef3-bb60-7cb8ab623af3 > > I don't have access to the broker side, so I am sorry, I

Re: CVE-2021-44228 – Log4j 2 Vulnerability

name to this kind of for name) One thing to add is that, we are currently working on upgrading log4j 1 to log4j 2 (KAFKA-9366 <https://issues.apache.org/jira/browse/KAFKA-9366>), and we'll make sure it upgrades to 2.15.0 or newer versions. Thank you. Luke On Sun, Dec 12, 2021 at 12:00

Re: Huge latency at consumer side ,testing performance for production and consumption

Hi Jigar, As Liam mentioned, those are necessary consumer initialization processes. So, I don't think you can speed it up by altering some timeouts/interval properties. Is there any reason why you need to care about the initial delay? If, like you said, the delay won't happen later on, I think the

Re: CVE-2021-44228 – Log4j 2 Vulnerability

Log4J exploit is getting a lot of publicity in my organization and a > > page to point our security team to would be very helpful. > > > > Brian > > > > Quoting Luke Chen : > > > > > Due to this vulnerability is quite critical and "popular"

Re: Kafka Streams - left join behavior

> know no consecutive inner-join result will happen. It's tricky to build > > though. > > > > I would recommend to wait and upgrade to 3.1 after it was releases. > > > > > > -Matthias > > > > On 11/30/21 12:59 AM, Luke Chen wrote: > > &g

Re: CVE-2021-44228 – Log4j 2 Vulnerability

://github.com/apache/kafka-site/pull/388. If > >> anyone > >> >> has more feedback, we can iterate on the PR. > >> >> > >> >> Thanks, > >> >> > >> >> Jun > >> >> > >> >> > >&g

Re: When will Kafka without ZooKeeper be GA?

Hi Christian, You can check this answer by Colin (Kafka PMC member): https://lists.apache.org/thread/r53hykwsb621rj37m5w1l1c7l71s6qc6 In short, it might need 6 month or a year. Thank you. Luke On Wed, Dec 15, 2021 at 2:30 AM Christian A. Mathiesen wrote: > Hi, > > In the quickstart guide on t

Re: Configuring topic level configuration MirrorMaker 2.0 in kafka v3.0

Hi Jigar, This is a known issue (KAFKA-13255 ) that we will be fixed in v3.2.0. I've asked if there is any workaround for this issue in the JIRA comments. Thank you. Luke On Wed, Dec 15, 2021 at 1:24 PM Jigar Shah wrote: > Hello, > I am trying

Re: When will the version 3.2.0 be released?

Hi David, Here's the release plan page: https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan Currently, we haven't had release date plan for v3.2.0, because v3.1.0 is still on-going. But if you check the release interval of each release previously, you can guess v3.2.0 should be

Re: Configuring topic level configuration MirrorMaker 2.0 in kafka v3.0

perties.exclude` in mirror maker config, to exclude the topic config properties that should not be replicated. In your case, it's "remote.storage.enable". Hope this helps. Thank you. Luke On Wed, Dec 15, 2021 at 9:13 PM Luke Chen wrote: > Hi Jigar, > > This

Re: Kafka Streams - start consuming from specific offset

Hi Miguel, > Is there a way to configure KafkaStreams or the consumer it uses to start from a specific offset? You can try to use `bin/kafka-streams-application-reset.sh` to reset the offset to a specific date time. REF: https://kafka.apache.org/30/documentation/streams/developer-guide/app-reset-

Re: Unable to start broker only config in Kraft mode (Kafka 3.0.0)

Hi Vamsee, >From the error stack trace, it is complaining the listener name: set in `controller.listener.names` cannot find the mapping security protocol. Ex: In https://github.com/apache/kafka/blob/trunk/config/kraft/broker.properties, we set the `controller.listener.names=CONTROLLER`, and in the

Re: Unable to start broker only config in Kraft mode (Kafka 3.0.0)

ller.listener.names' altogether > on this node but it fails to start with another exception. > > To summarize, if a minimal broker only and controller only config examples > can be added to the Kraft README they would be awesome 👍 > > > On Sat, Dec 18, 2021 at 4:59, Luke C

Re: "bin/kafka-configs.sh" is not working when used with Kraft

Hi Jacob, Answering your question below: > For example from the doc this should work > bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe > But it doesn This is working as designed. It's because you didn't alter any dynamic configuration for t

Re: kafka and zookeeper

Hi Yonghua, Thanks for asking. Currently, the community is still working in progress for the feature to remove zookeeper (aka. Kraft mode). You can run the Kraft mode in testing environment now, but not in production env. There's no a specific timeline for this feature to be completed. For more d

Re: CVE-2021-44228 – Log4j 2 Vulnerability

in the log4j.properties file. Does it > mean we are safe ? > > Thanks, > Dhirendra. > > -Original Message- > From: Brian Rickabaugh > Sent: Wednesday, December 15, 2021 8:04 AM > To: users@kafka.apache.org > Subject: Re: CVE-2021-44228 – Log4j 2 Vulnerability >

Re: Kafka connect JMS sink connector performance

Hi Tilak, > While using the connector we observed that, for each call to the `put` method in our `SinkTask` implementation, there is some delay between calls which is in the range of [200ms to 1sec]. Yes, that makes sense! As you should have known, the sinkTask is also a Kafka consumer, which try

Re: Enable TLS Authentication + Enable External Access using LoadBalancer

Hi Wei Yang, > On Kafka cluster side, how to configure *advertised.listeners* for external access? All 3 LoadBalancer IPs + port, or any 1 LoadBalancer IP + port? Since you have one LoadBalancer per broker, you should set *advertised.listeners *one LoadBalancer IP + port. You can check this good

Re: Consumer failure after rolling Broker upgrade

not occur with 2.5.1 and 2.7.0 Clients. > This make me suspect that > https://issues.apache.org/jira/browse/KAFKA-10793 introduced this issue. > > Regards, James. > > On 24/11/2021, at 14:35, James Olsen wrote: > > Luke, > > We did not upgrade to resolve the issue

Re: Log4j 2.x preview for Kafka

Hi Deepak, As Israel mentioned, the core Apache Kafka components do not have any dependencies on log4j2. The only CVE that core Apache Kafka got impacted is CVE-2021-4104. You can check the official announcement for the mitigation methods here: https://kafka.apache.org/cve-list For the log4j 2.x

Re: Log4j 2.x preview for Kafka

ation. Just one more query: > > Since, we do not use the JMS Appender so do we need to apply the > mitigation mentioned in the below link for CVE-2021-4104. > https://kafka.apache.org/cve-list > > Regards, > Deepak > > -Original Message- > From: Luke Chen >

Re: Unit Test falling for Kafka 2.8.0

Hi Mohammad This is a known issue because of JFrog sunset the Bintray repo. We've fixed this issue in v3.0 (PR: https://github.com/apache/kafka/pull/10056). You should build/run tests successfully in v3.0. If you can't change to v3.0, then you can refer to the PR above to fix it. But we're not sur

Re: [External] : Re: Unit Test falling for Kafka 2.8.0

gt; > >- Gradle 7.3.3 >- >- java version "11.0.13" 2021-10-19 LTS >- Java(TM) SE Runtime Environment 18.9 (build 11.0.13+10-LTS-370) >- Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.13+10-LTS-370, >mixed mode) >- >- Scala code ru

Re: [External] : Re: Unit Test falling for Kafka 2.8.0

> > * Try: > Run with --stacktrace option to get the stack trace. Run with --info or > --debug option to get more log output. Run with --scan to get full insights. > > * Get more help at https://help.gradle.org > > Deprecated Gradle features were used in this build, making it i

Re: Kafka Streams - one topic moves faster the other one

Hi Miguel, Yes, the grace period is the solution to fix the problem. Alternatively, you can try to set a higher value for "max.task.idle.ms" configuration, because this is some kind of out-of-order data. Let's say, A topic has 1 record per second (fast), B topic has 1 record per minute (slow). Yo

Re: About the order of messages when a single consumer is consuming messages from multiple partitions

Hi Roger, > Can anything be said about the order of the messages consumed by my consumer? Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written. So, if you create 4 consumers in a group, to consume fro

Re: [VOTE] 3.1.0 RC0

Hi David, I've done: 1. Verified checksums and signatures 2. Ran quick start using scala 2.13 3. Browse the java doc All looks good. +1 (non-binding) Thanks for running the release. Luke On Fri, Jan 7, 2022 at 11:38 PM Israel Ekpo wrote: > So far we have 1 binding +1 vote and 4 non-binding

Re: Questions around static membership partition assignment

Hi Jun The goal of static membership, is to hold the rebalance when there's consumer dropped (before session timeout). For K8s, it's good because when the pods are broken (or during upgrade), it'll kill the pod and bring a new one up to replace the old one. In this case, we don't want the consumer

Re: Kafka Log4j2.x upgrade plan

Hi Deepak, So far, we don't have an ETA for log4j2. Please check this discussion: https://issues.apache.org/jira/browse/KAFKA-9366 Thank you. Luke On Fri, Jan 21, 2022 at 1:57 PM Deepak Jain wrote: > Hi Luke, > > We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to the > Log4j

Re: [ANNOUNCE] Apache Kafka 3.1.0

UN-HAO TANG, Cheng Tan, Chia-Ping Tsai, > > Chris Egerton, Christo Lolov, Colin P. McCabe, Cong Ding, Daniel > > Urban, David Arthur, David Jacot, David Mao, Dmitriy Fishman, Edoardo > > Comar, Ewen Cheslack-Postava, Greg Harris, Guozhang Wang, Igor Soarez, > > Ismael Juma,

Re: Is there an equivalent of a "synchronous subscribe"?

Yes, there is no such "synchronous subscribe" method in Kafka. Consumer group will start to work after `poll` is called. Inside Kafka, there are util methods like `awaitAssignment` or `awaitRebalance` to wait for the initialization completed for tests, which is checking with `poll` You can refer t

  1   2   3   >