Re: Kafka as a broadcast tool?

You can create a new topic for each of your client which comes online. If your client has some sort of identifier like SessionId, you can create a topic with sessionId as 'prefix/suffix' and send messages to that client. Once your client closes / goes offline, delete that topic using AdminClient.

Re: Kafka log4j appender

You can find it here https://logging.apache.org/log4j/2.x/manual/appenders.html#KafkaAppender On Sat, Nov 4, 2017 at 1:29 AM, Preston, Dale < dale.pres...@conocophillips.com> wrote: > Did the log4j appender get deprecated? I've spent a good amount of time > on google and don't find any such anno

Broker Already Registered Error

- 0.10.2.1 and Zookeeper version - 3.4.9 Does these session errors are fixed in the latest version (1.0.0) / What are the pre-cautionary steps to take to avoid these errors ? Regards, Kamal C

Re: Broker Already Registered Error

With the `zookeeper-shell.sh` script, I have checked the path of `/brokers/ids`, it showed only the broker id which was un-affected. On Fri, Nov 24, 2017 at 12:49 PM, Kamal wrote: > Hi Kafka Users, > > In our production cluster, we have faced the below error in 2 out of 3 > br

Re: Ambiguous entry in docs

A consumer can read data from multiple topics. The above line describes the usage of topic regex in which an existing consumer can read data from newly (yet-to) created topics. For more details, see https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscrib

Re: connecting to 2 different clusters with different sasl realms in a single process

I'd like to point out that you can pass multiple jaas config within a jvm using `sasl.jaas.config` in producer & consumer config. On Mon, Feb 19, 2018 at 8:10 PM, Michal Hariš wrote: > Hi all, > > I have one cluster with kerberos authenticator and another with simple > authenticator. I need to b

How to configure KAFKA outborne ports

Hi, We are running KAFKA behind firewall. How to configure the KAFKA outborne ports? Regards, Kamal C -- Open WebMail Project (http://openwebmail.org) -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

how to configure KAFKA behind firewall

tcp0 0 :::54300:::* LISTEN 6519/java tcp0 0 :::9092 :::* LISTEN 6519/java I need to configure all ports since we have to run KAFKA and ZooKeeper behind firewall. Thanks for your time! Cheers, Kamal C

Kafka Cluster Issue

ts to throw Connect Exception continuously and tries to connect with the dead node (not all producers). Is there any configuration available to avoid this exception ? Regards, Kamal C

Re: Kafka Cluster Issue

Any comments on this issue? On Sat, May 2, 2015 at 9:16 AM, Kamal C wrote: > Hi, > We are using Kafka_2.10-0.8.2.0, new Kafka producer and Kafka Simple > Consumer. In Standalone mode, 1 ZooKeeper and 1 Kafka we haven't faced any > problems. > > In cluster mode, 3 ZooKee

Re: Kafka Cluster Issue

gt; > On 5/2/15, 11:01 PM, "Kamal C" wrote: > > >Any comments on this issue? > > > >On Sat, May 2, 2015 at 9:16 AM, Kamal C wrote: > > > >> Hi, > >> We are using Kafka_2.10-0.8.2.0, new Kafka producer and Kafka Simple > >> Consum

Re: Kafka Cluster Issue

This is resolved. As I missed host entry configuration in my infrastructure. On Mon, May 4, 2015 at 10:35 AM, Kamal C wrote: > We are running ZooKeeper in ensemble (Cluster of 3 / 5). With further > investigation, I found that the Connect Exception throws for all "inflight&q

How Producer handles Network Connectivity Issues

leader node by throwing connection refused exception. I understand that when there is a node failure leader gets switched. Why it's not switching the leader in this scenario ? -- Kamal C

Re: how to wait next log message on the partition queue

It won't throw OffsetOutOfRange error when you pass the latest offset to the fetch request, the resulting fetch response message set would be empty. You can wait for message to be available either manually [or] by configuring *maxWait* in fetch request. On Mon, May 25, 2015 at 3:50 PM, Nipur Patod

Re: How Producer handles Network Connectivity Issues

27;t connect to the broker, how would it even report that it has a > connectivity issue? > > -Ewen > > On Mon, May 25, 2015 at 10:05 PM, Kamal C wrote: > > > Hi, > > > > I have a cluster of 3 Kafka brokers and a remote producer. Producer > > started t

Re: kafka producer error

Refer the mail list http://qnalist.com/questions/6002514/new-producer-metadata-update-problem-on-2-node-cluster https://issues.apache.org/jira/browse/KAFKA-1843 On Wed, Sep 9, 2015 at 7:37 AM, Shushant Arora wrote: > Hi > > I have a kafka cluster with 3 brokers. I have a topic with ~50 partitio

Re: Kafka + ZooKeeper on the same hardware?

It's a single point of failure. You may lose high-availability. On Thu, Jan 14, 2016 at 4:36 PM, Erik Forsberg wrote: > Hi! > > Pondering how to configure Kafka clusters and avoid having too many > machines to manage.. Would it be recommended to run say a 3 node kafka > cluster where you also ru

Re: Kafka + ZooKeeper on the same hardware?

Yes, it can sustain one failure. Misunderstood your question.. On Thu, Jan 14, 2016 at 5:14 PM, Erik Forsberg wrote: > > > On 2016-01-14 12:42, Kamal C wrote: > >> It's a single point of failure. You may lose high-availability. >> > > In this case I would li

Re: seekToBeginning doesn't work without auto.offset.reset

Cody, Use ConsumerRebalanceListener to achieve that, ConsumerRebalanceListener listener = new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection partitions) { } @Override public void onPartitionsAssigned

How to assign all the partitions of a topic to consumer ?

topic? --Kamal

Re: How to assign all the partitions of a topic to consumer ?

Thanks Jason! It worked. I've missed it. On 17-Mar-2016 2:00 AM, "Jason Gustafson" wrote: > Have you looked at partitionsFor()? > > -Jason > > On Wed, Mar 16, 2016 at 4:58 AM, Kamal C wrote: > > > Hi, > > > > I'm using the new con

Re: Kafka shutdown gracefully

Don't use `kill -9 PID`. Use `kill -s TERM PID` - sends a signal to the process to end, and will trigger any cleanup routines before exiting. Since the output of the `ps` command used by kafka-server-stop.sh exceeds 4096 characters. It shows "No kafka server to stop" On Thu, Jul 6, 2017 at 3:25 A

Re: Kafka Producer Errors

for the KafkaProducer. -- Kamal On Sat, Aug 12, 2017 at 12:19 AM, Saladi Naidu < naidusp2...@yahoo.com.invalid> wrote: > Looks like some of the fonts/characters did not come through, am > re-sending with correct details > We have recently upgraded our Kafka cluster from 0.8.2 to

Re: Log flush

es to disk to the OS Kernel. This is done to maintain faster read & write performance. Kafka doesn't rely on `fsync` for message durability. Instead, it relies on it's own replicated change-logs located in other brokers. -- Kamal On Wed, Aug 16, 2017 at 12:45 AM, Jakes John w

Re: Why does kafka-consumer-groups show the topics written to too in its describe

`through` = `to` + `stream` operation. So, the consumer-groups command showing the "fname-stream" topic. Use `to`, if you just want to write the output to the topic. -- Kamal On Mon, Aug 21, 2017 at 12:05 PM, Sachin Mittal wrote: > Folks any thoughts on this. > Basically I

Re: Reduce Kafka Client logging

add this lines at the end of your log4j.properties, log4j.logger.org.apache.kafka.clients.producer=WARN On Thu, Sep 7, 2017 at 5:27 PM, Raghav wrote: > Hi Viktor > > Can you pleas share the log4j config snippet that I should use. My Java > code's current log4j looks like this. How should I add

KTable-KTable Join Semantics on NULL Key

or all the 3 types of joins on receiving a record with NULL value. [1]: https://cwiki.apache.org/confluence/display/KAFKA/ Kafka+Streams+Join+Semantics#KafkaStreamsJoinSemantics-KTable-KTableJoin -- Kamal

Re: kafka user

Please follow the instructions listed here https://kafka.apache.org/contact You have to send a mail to 'users-subscr...@kafka.apache.org' to subscribe to user list. On Tue, Oct 31, 2017 at 1:06 PM, Karthigeyan wrote: > Hi , > > pls add to the user group. > > Thanks , > > Karthigeyan > >

Re: Kafka Zookeeper Connection

You also have to update the property `zookeeper.session.timeout.ms On Sat, Feb 24, 2018 at 11:25 PM, Ted Yu wrote: > Please take a look at maxSessionTimeout under: > http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin. > html#sc_advancedConfiguration > > On Sat, Feb 24, 2018 at 9:46 AM, Soheil

Re: 答复: [ANNOUNCE] New Committer: Dong Lin

Congratulations, Dong! On Thu, Mar 29, 2018 at 7:14 AM, Manikumar wrote: > Congrats, Dong! > > On Thu, Mar 29, 2018 at 6:45 AM, Tao Feng wrote: > > > Congrats Dong! > > > > On Wed, Mar 28, 2018 at 5:15 PM Dong Lin wrote: > > > > > Thanks everyone!! > > > > > > It is my great pleasure to be par

Async Consumer: atleast once delivery

pproach but it's not scalable. How to track the offsets of processed records? --Kamal

Re: Consumer Client - How to simulate heartbeats ?

Hi Florian, This may be helpful https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java --Kamal On Fri, Apr 15, 2016 at 2:57 AM, Jason Gustafson wrote: > Hi Florian, > > It's actually OK if processin

Re: Consumer Client - How to simulate heartbeats ?

When a new consumer joins to the group, it should start to read data from where the other consumer left. --Kamal On Mon, Apr 18, 2016 at 6:58 PM, Florian Hussonnois wrote: > Thank you very much, the example is really helpful. > > My last question is : Why is it necessay to seek the

Re: Consumer Client - How to simulate heartbeats ?

Yes, you're right. No need to seek the offsets in onPartitionsAssigned method internally KafkaConsumer handles it. Thanks for sharing this with me. I'll update it. --Kamal On Mon, Apr 18, 2016 at 7:29 PM, Florian Hussonnois wrote: > Yes, but the ConsumerRebalanceListener is op

Re: What makes a message key mandatory and how to turn it off?

Can you describe your topic configuration using the below command ? *sh kafka-topics.sh --zookeeper localhost:2181 --topic --describe* Key for a record is mandatory only for compacted topics. --Kamal On Wed, May 4, 2016 at 2:25 PM, I PVP wrote: > HI all, > > What makes a me

Re: What makes a message key mandatory and how to turn it off?

Yes. Use *log.cleanup.policy=delete* if you don't want to compact topics. Reference: https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction On Wed, May 4, 2016 at 3:24 PM, I PVP wrote: > Kamal, > > Could the log.cleanup.policy=compact on server.properties be the > c

Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

Andy, Kafka 0.9.0 server supports the previous versions of the clients (0.8.2, 0.8.1..). But, new clients won't work properly with the older version of Kafka server. You should upgrade your server / broker first. --Kamal On Fri, May 20, 2016 at 10:58 PM, Andy Davidso

Fwd: Questions about Kafka Scripts

Hi, A. What is the usage of `kafka-replica-verification.sh` script ? I don't find any documentations about it in [1] and [2]. I have a topic `test` with 10 partitions. Ran the above script, it continuously prints the below results. [kamal@tcltest1 bin]$ sh kafka-replica-verification.sh -

Re: Kafka Consumer poll

> A.SathishKumar > [1]: https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html -- Kamal

Re: Kafka consumers in cluster

Implement your own custom `org.apache.kafka.clients.consumer.internals.PartitionAssignor` and assign all the subscribed partitions to the first consumer instance in the group. See 'partition.assignment.strategy' config in the consumer configs [1] [1]: http://kafka.apache.org/documentation.html#ne

Re: Kafka consumers in cluster

Yes, it gets called on every re-balance. -- Kamal On Thu, Aug 4, 2016 at 11:24 PM, sat wrote: > Hi Kamal, > > Thanks for your prompt response. Does our custom partition assignor gets > called during every rebalancing. > > Thanks and Regards > A.SathishKumar > > >

Pausing a partition in Kafka Consumer

this consumer it starts to read data from that partition. How to pause a partition even if re-balance occurs ? Regards, Kamal C

Re: producer can't push msg sometimes with 1 broker recoved

Reduce the metadata refresh interval 'metadata.max.age.ms' from 5 min to your desired time interval. This may reduce the time window of non-availability broker. -- Kamal

Re: producer can't push msg sometimes with 1 broker recoved

: 4 sent [2016-09-27T15:18:19,182] KafkaProducer: main(): Batch : 5, Sending the record with key : 0 - Kamal On Mon, Sep 26, 2016 at 8:53 AM, FEI Aggie wrote: > Kamal, > Thanks for your response. I tried testing with metadata.max.age.ms > reduced to 10s, but the behavior not changed, and

Log Compaction: How Key comparison works?

ta-type ? 2. K1 == K1 returns true ? 3. K1.equals(K1) returns true ? (Even, if custom equals() and hashcode() implemented) [1]: https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction - Kamal

Re: How to keep consumers alive without polling new messages

You can refer this example[1] [1]: https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java - Kamal On Wed, Sep 28, 2016 at 11:33 AM, Vincent Dautremont < vincent.dautrem...@olamobile.com> wrote: > I had

Re: programmatic way to check for topic existence?

Ben, You can list all the available topic information and do a simple look up from the returned list. Map> topics = consumer.listTopics(); topics.contains () - isn't enough? -- Kamal On 25 Oct 2016 22:56, "Ben Osheroff" wrote: > We won't proceed in the face of a

Re: Facing Duplication in consumer

If the consumer is idle (not doing any commit) for more than a day, the offsets will be wiped out. Rf. offsets.retention.minutes property in Kafka documentation. On Tue, May 29, 2018 at 9:49 AM, Shantanu Deshmukh wrote: > Which Kafka version? > > On Mon, May 28, 2018 at 9:09 PM Dinesh Subramani

Re: How set properly infinite retention

log.retention.ms = 9223372036854775807 (Long.MAX_VALUE) On Mon, Jul 30, 2018 at 3:04 PM David Espinosa wrote: > Hi thanks a lot for the reply. > > The thing is that I need compaction to delete some messages (for GDPR > purposes), and for that I need the log cleaner to be enabled (with > policy=

Re: Very long consumer rebalances

In v0.10.0.1, consumer heartbeat background thread feature is not available. Lot of users faced similar errors. So, KIP-62 is proposed. You have to update your Kafka version

Re: measuring incoming bytes

In Yammer metrics, the 15MinuteRate provided by the Meter is not a pure average for the last 15 minutes. It's a Exponential Weighted Moving Average. The value is calculated by taking 36% of weighted average from the broker start time and 64 % of weighted average from the last 15 minutes. I initial

Re: Restart process after adding control.plane.listener.name config in Kafka 2.2.0

Migration plan for this change is mentioned in the KIP-291 . Take a look at the migration plan. Migration plan: 2 rounds of rolling upgrades are needed to pick up the >

Re: Best Practice Scaling Consumers

1. Yes, you may have to overprovision the number of partitions to handle the load peaks. Refer this document to choose the no. of partitions. 2. KIP-429

Re: Kafka upgrade process details

Hi, In Kafka v2.1.0, the OffsetCommit Request/Response schema version is changed to v4 for the *__consumer_offsets* topic. If you upgrade Kafka to v2.1.0 & higher and changed the inter.broker.protocol version to 2.1, then you cannot revert back to older versions as it doesn't know how to read the

Re: Kafka transaction between 2 kafka clusters

MirrorMaker 2.0 stores the offsets of one cluster in another. So, you can read the offsets from the same cluster once this KIP is implemented. https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RemoteClusterUtils On Fri, May 10, 2019 at 12:29 PM Em

Re: KeeperException

Those are not errors. See the logs are logged in INFO mode. https://stackoverflow.com/a/48067058/3209010 On Wed, May 15, 2019 at 10:44 AM Gagan Sabharwal wrote: > Hi team, > > Any pointers on the same ? > > Regards > Gagan > > On Mon, May 13, 2019 at 11:24 AM Gagan Sabharwal > wrote: > > > Hi

Re: Kafka delaying message

If you have a dedicated topicPartition for delayed messages, you can pause that partition for 15 min to avoid blocking the polling thread. On Thu, May 2

Re: [VOTE] 2.3.0 RC3

+1 (non-binding) * Ran unit and integration test on 2.11 and 2.12 * Verified quick start * Ran internal apps on the 3 node cluster On Thu, Jun 20, 2019 at 3:33 AM Colin McCabe wrote: > Hi all, > > We discovered some problems with the second release candidate (RC2) of > 2.3.0. Specifically, KAF

Re: list of pattern processed topic list

You can use the KafkaConsumer#assignment() method to get all the assigned topic-partitions for that consumer instance. But, you've to periodically call poll method to get the latest assignment which may return records. This shortcoming is actively discussed in the below threads. https://mail-archi

Re: [ANNOUNCE] New committer: John Roesler

Congrats John! On Wed, Nov 13, 2019 at 7:57 AM Dong Lin wrote: > Congratulations John! > > On Tue, Nov 12, 2019 at 1:56 PM Guozhang Wang wrote: > > > Hi Everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, > John > > Roesler. > > > > John has been contributing

Re: Max poll interval and timeouts

Hi Ryan, The maxPollInterval waits for at-most the given time duration and returns ASAP even if a single record is available. If you want to collect data once 30-45 minutes, better to use the Consumer with `assign` mode and poll for records once in 30 minutes. If you're using the consumer with `

Re: MirrorMaker2 - uneven loadbalancing

Hi Peter, Not sure this is what you're looking for - https://issues.apache.org/jira/browse/KAFKA-9352 On Mon, Mar 23, 2020 at 11:37 PM Ryanne Dolan wrote: > Thanks Peter for running this experiment. That looks sorta normal. It looks > like Connect is deciding to use 10 total tasks and doesn't c

Re: Max poll interval and timeouts

24, 2020 at 11:56 PM Ryan Schachte wrote: > Don't I lose consumer group coordination with assign? > > On Mon, Mar 23, 2020 at 11:49 PM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Hi Ryan, > > > > The maxPollInterval waits fo

Re: Max poll interval and timeouts

With group coordination protocol, you only have to increase the ` max.poll.interval.ms` / `max.poll.records`. Ignore the above messages. Consumer heartbeats are processed in a separate thread. On Wed, Mar 25, 2020 at 2:35 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote:

Re: Offset commit consistently timing out in transaction

Timeout in offset commit request has been fixed recently. This issue seems to be more of KAFKA-8334 . On Mon, Jan 4, 2021 at 3:53 PM Kindernay Oliver wrote: > Hello, > > we are experiencing problems with offset commit timing out on brokers. > Th

Re: Release plan required for version 3.5.1

Hi Sahil, Apache Kafka 3.5.1 is already released: https://kafka.apache.org/downloads On Wed, Jul 26, 2023 at 9:08 AM Sahil Sharma D wrote: > Gentle reminder-2 > > -Original Message- > From: Sahil Sharma D > Sent: 12 July 2023 09:51 AM > To: users@kafka.apache.org > Subject: RE: Release

Re: [VOTE] 3.6.0 RC2

+1 (non-binding) 1. Built the source from 3.6 branch in scala 2.12 and 2.13 2. Ran all the unit and integration tests. 3. Ran quickstart and verified the produce-consume on a 3 node cluster. 4. Verified the tiered storage functionality with local-tiered storage. Thanks, Kamal On Mon, Oct 2

Re: Script to delete list of topics

for topic in `cat /tmp/topics.txt`; do echo $topic; sh kafka-topics.sh --bootstrap-servers localhost:9092 --topic $topic --delete; done # you can also delete the topics by wildcard sh kafka-topics.sh --bootstrap-servers localhost:9092 --topic abc.* --delete On Tue, Jan 23, 2024 at 11:50 AM sunil c

Re: Kafka followers with higher leader epoch than leader

entries in those files in Line 2. 4. Remove the `.kafka_cleanshutdown` marker file. 5. Start the node and trigger preferred leader election to elect back the same node as leader 6. Then, the follower will be able to sync up with the leader. -- Kamal On Tue, Mar 19, 2024 at 6:06 PM Karl Sorensen

Re: How do we usually handle Node disconnected issue for kafka producer

Hi Sachin, Why do you want to change the default settings? If the connection is open and unused, then it is fair to close the connection after the timeout and reopen it when required. On Fri, May 3, 2024 at 1:06 PM Sachin Mittal wrote: > Hi, > I am using a Kafka producer java client by vert.x f

Re: Network partition leaves topic-partition leader as sole ISR despite min.isr=2 and producer acks=all settings

Hi Sabit, Thanks for reporting the issue! This is the last standing replica problem and is being fixed in KIP-966. You can go through the below blog to understand it in detail: https://jack-vanlightly.com/blog/2023/8/17/kafka-kip-966-fixing-the-last-replica-standing-issue#:~:text=Rule%20number%20

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

ggest you upgrade to KRaft. On Sat, Nov 2, 2024 at 4:15 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Jianfeng, > > As Luke already mentioned, the feature was not tested in ZK mode. > If you still want to enable it in the ZK mode, then: >

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

red/tested: (eg) replica down during disablement, replica was down during disablement followed by enablement, and so on. Read the discussion and voting thread for more details. Thanks, Kamal On Wed, Oct 30, 2024 at 12:56 PM Luke Chen wrote: > Hi Jianfeng, > > The reason we don't

Re: Apache Kafka integration using Apache Camel

Can you enable DEBUG logs ? It'll be helpful to debug. -- Kamal On Mon, Jan 9, 2017 at 5:37 AM, Gupta, Swati wrote: > Hello All, > > Any help on this would be appreciated. > There seems to be no error. Does it look like a version issue? > > I have updated my

Re: Kafka Producer - Multiple broker - Data sent to buffer but not in Queue

> bootstrap.servers = , Is your bootstrap.servers configuration is correct ? You have specified port `9091`, but running the GetOffsetShell command on `9094` On Wed, Apr 19, 2017 at 11:58 AM, Ranjith Anbazhakan < ranjith.anbazha...@aspiresys.com> wrote: > Unfortunately, there is no specifi

Re: Exception in thread "main" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries

ttp://en.wikipedia.org/wiki/Environment_variable#Microsoft_Windows>\system32\drivers\etc\hosts put an entry of your { serverIphostname} Regards, Kamal C. On Sat, Aug 2, 2014 at 11:52 AM, wrote: > Hi Team, > > I am trying with Kafka client on Windows 7 64bit -corporate pc which is >