Re: Question: Data Loss and Data Duplication in Kafka

Experimenting with kafka myself, and found timeouts/batch expiry (valid and invalid configurations), and max retries also can drop messages unless you handle and log them gracefully. There are also a bunch of org.apache.kafka.common.KafkaException hierarchy exceptions some of which are thrown for v

Auto offset commit failed

Hi All, My kafka cluster is kafka0.10.0. I have found two reasons for Auto offset commit failed in the log file. One is Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member, and another is Commit offsets failed with retriable exc

Re: How distributed countByKey works in KStream ?

Hi Tommy, yes, you do understand Kafka Streams correctly. And yes, for shuffling, na internal topic will be created under the hood. It should be named "-something-repartition". I am not sure, why it is not listed via bin/kafka-topics.sh The internal topic "-counts-changelog" you see is created to

Re: a broker is already registered on path /brokers/ids/1

I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually. Since you're on the 3.5 branch, you may wa

Re: kafka broker is dropping the messages after acknowledging librdkafka

Hi Jun, Yes, the data is lost during leader broker failure. But the leader broker failed due to zookeeper session expiry. GC logs doesn't show any error/warns during this period. Its not easy reproduce. during long run (>12hrs) with 30k msg/sec load balanced across 96 partitions, some time in bet

Re: How distributed countByKey works in KStream ?

Does Kafka support hidden topics ? (Since all the topics infos are stored in ZK, this probably not the case ) On Tue, Aug 30, 2016 at 5:58 PM, Matthias J. Sax wrote: > Hi Tommy, > > yes, you do understand Kafka Streams correctly. And yes, for shuffling, > na internal topic will be created under

Re: kafka-streams project compiles using maven but failed using sbt

Got it. Thanks Michael. On Tue, Aug 30, 2016 at 2:31 AM, Michael Noll wrote: > Most probably because, in your build.sbt, you didn't enable the > -Xexperimental compiler flag for Scala. This is required when using Scala > 2.11 (as you do) to enable SAM for Java 8 lambda support. Because this >

Re: How distributed countByKey works in KStream ?

No. It does not support hidden topics. The only explanation might be, that there is no repartitioning step. But than the question would be, if there is a bug in Kafka Streams, because between map() and countByKey() repartitioning is required. Can you verify that the result is correct? -Matthias

Re: a broker is already registered on path /brokers/ids/1

Setup is 3 participants + 1 observer. We used 3.5.0 for dynamic configs. What happens when a system forced killed the zookeeper process and it did not have time to clean up ephemeral? Does it recover in that scenario? Thanks. Sent from my iPad > On Aug 30, 2016, at 8:53 AM, Flavio Junqueira

ApacheCon Seville CFP closes September 9th

It's traditional. We wait for the last minute to get our talk proposals in for conferences. Well, the last minute has arrived. The CFP for ApacheCon Seville closes on September 9th, which is less than 2 weeks away. It's time to get your talks in, so that we can make this the best ApacheCon yet. I

How do I remove myself from mailing list?

I've gone through the unsubscribe process more times than I can count, but I'm still getting emails. How can I unsubscribe? Who is an admin? This message may contain information that is privileged or confidential. If you received this transmission in error, p

RE: How do I remove myself from mailing list?

I have done the same several times, but I still get these emails. -Original Message- From: Spencer Owen [mailto:so...@netdocuments.com] Sent: Tuesday, August 30, 2016 8:33 AM To: users@kafka.apache.org Subject: How do I remove myself from mailing list? I've gone through the unsubscribe p

Re: a broker is already registered on path /brokers/ids/1

Is the zookeeper process you're referring to the server or the client (broker in the case of Kafka)? If you're referring to the server, then the ensemble recovers from the disk state of the servers. If it is a follower, then there is nothing to recover, the leader already has all the necessary

Re: a broker is already registered on path /brokers/ids/1

Kafka broker in my case Printing e-mails wastes valuable natural resources. Please don't print this message unless it is absolutely necessary. Thank you for thinking green! Sent from my iPhone > On Aug 30, 2016, at 1:19 PM, Flavio Junqueira wrote: > > Is the zookeeper process you're referrin

Re: consumer reread messages

You can you seek. https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seek(org.apache.kafka.common.TopicPartition,%20long) Or if you want to re read the entire log use a different consumer group name and use auto.offset.reset=earliest On Tue, Aug 30, 2016

Kafka unable to process message

Hi there, What does the below error mean and how to avoid this? I see this error one of the kafkaServer.out file when other broker is down. And not able to process any message as we see o.a.k.c.c.i.AbstractCoordinator - Issuing group metadata request to broker 5 from application log [2016-08

Re: How do I remove myself from mailing list?

count me out too! ;-) On Tue, Aug 30, 2016 at 12:54 PM, Heath Ivie wrote: > I have done the same several times, but I still get these emails. > > -Original Message- > From: Spencer Owen [mailto:so...@netdocuments.com] > Sent: Tuesday, August 30, 2016 8:33 AM > To: users@kafka.apache.org

Re: How do I remove myself from mailing list?

Just to confirm: after you unsubscribe by sending an email to users-unsubscr...@kafka.apache.org, you should receive an email titled "confirm unsubscribe from users@kafka.apache.org", which you need to reply to in order for the unsubscribe to take effect. Did you folks do that? On Tue, Aug 30, 201

Re: How do I remove myself from mailing list?

Shikhar, I never got the "confirm unsubscribe from..." follow up email, hence the problem. Phil On Tue, Aug 30, 2016 at 6:49 PM, Shikhar Bhushan wrote: > Just to confirm: after you unsubscribe by sending an email to > users-unsubscr...@kafka.apache.org, you should receive an email titled > "c

What should my Consumer connect to Kafka/Zookeeper

Hi Everyone, I am trying to get some information around what my consumer should connect to. According to the docs sending messages should be directly to the Kafka broker and consuming messages should be from Zookeeper? I am really after a reason this is the case as I am aware that you can just

Re: What should my Consumer connect to Kafka/Zookeeper

Your consumers/producers will push/pull messages directly from kafka. More specifically, to/from the broker that is responsibile for the partition assigned to the consumer or corresponding to the published message key. Others should correct me if I am wrong, but zookeeper was used to store informa

Re: Monitoring the max lag of a kafka streams application.

Hello Rohit, As you are already aware, monitoring kafka streams is no difference than monitoring kafka producers / consumers. So you can just monitor on its embedded consumer's "records-lag-max" metric, which gets recorded whenever the consumer received the fetch response. As for your application

Re: Monitoring the max lag of a kafka streams application.

Hi Rohit, Just for clarification, as stated in the java doc, metricChange "is called whenever a metric is updated or added". It is not the function when a metric is recorded; in fact, the metrics collection is in the "pull" model, where the implemented reporters can fetch the current calculated va

Re: Kafka Streaming Join for range of gps coordinates

Hello Farhon, I think your idea about KStream-KTable join is a good approach with some tweaks, more specifically: 1. Model your rider request as a normal record stream with the combo key of (latitude, longitude). 2. Model your driver location as an ever-updating table with the combo key of (lati

RE: handling generics in Kafka Scala

noob with Scala so Im looking for an experienced answer ConsumerGroupCommand.scala //private def createNewConsumer(): KafkaConsumer[String, String] = {//private def createNewConsumer(): KafkaConsumer[K extends java.util.ArrayList[Byte],V extends java.util.ArrayList[Byte]] = {private def creat

Re: How distributed countByKey works in KStream ?

Tried the word count example as discussed, the result in wc-out is wrong: a 1 > b 1 > a 1 > b 1 > c 1 The expected result should be: a 2 > b 2 > c 1 Kafka version is 0.10.0.1 On Tue, Aug 30, 2016 at 10:29 PM, Matthias J. Sax wrote: > No. It does not support hidden topics. > > The only exp

How to resolve WARN messages in kafka.server.ReplicaFetcherThread$FetchRequest: No response received within 30000 ms and Connection to xxx was disconnected

Hi, We have been using kafka_2.11-0.9.0.1. We’re seeing the following 2 warning messages in kafka.server.ReplicaFetcherThread whenever there is large read by spark. It’s always accompanied by Shrinking ISR and Expanding ISR messages. It seems that Spark is still receiving data. However, it’s unc

kafka.common.ConsumerRebalanceFailedException: can't rebalance after 4

HI all; Im using 0.8 consumer with kafka 0.10.0.1. When i run my consumer app (within wildfly 10.0.x) Im getting follwoing exception and consumer is not listning any messages; I tried increasing *"rebalance.backoff.ms ", "zookeeper.session.timeout.ms