Re: KTable send old values API

Dmitry, I think your use case is similar to the one I described in the link below (discussion in the kafka-dev mailing list): http://search-hadoop.com/m/uyzND1rVOQ12OJ84U&subj=Re+Streams+TTLCacheStore Could you take a quick look? -Michael On Wed, Feb 22, 2017 at 12:39 AM, Dmitry Minkovsky w

[ANNOUNCE] Apache Kafka 0.10.2.0 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.10.2.0. This is a feature release which includes the completion of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull requests merged. All of the changes in this release can be found in the release n

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Thanks Ewen for driving the release! Guozhang On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.2.0. This is a feature release which includes the completion > of 15 KIPs, over 200 bug fixes and

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Thanks Ewen for driving this. On Wed, Feb 22, 2017 at 12:40 AM, Guozhang Wang wrote: > Thanks Ewen for driving the release! > > Guozhang > > On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava > > wrote: > > > The Apache Kafka community is pleased to announce the release for Apache > > Kafk

[KIP-94] SessionWindows - IndexOutOfBoundsException in simple use case

Hello, I apologies with Matthias since I posted yesterday this issue on the wrong place on github :( I'm trying a simple use case of session windowing. TimeWindows works perfectly, however as I replace with SessionWindows, this exception is thrown: Exception in thread "StreamThread-1" org.apache

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Great news, thanks for driving the release Ewen! And thanks to all the contributors who made it possible. Ismael On 22 Feb 2017 8:33 am, "Ewen Cheslack-Postava" wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.2.0. This is a feature release which in

Re: [KIP-94] SessionWindows - IndexOutOfBoundsException in simple use case

Hi Marco, Did you run this example with the same store name using TimeWindows? It looks to me that it is trying to restore state from the changelog that has been used with TimeWindows. The data in the topic will be incompatible with SessionWindows as the keys are in a different format. You'll eit

Re: Heartbeats while consuming a message in kafka-python

Hi, thank you both for responses. I forgot to mention which python client I'm using, so let me fix that first: Currently I'm using https://github.com/dpkp/kafka-python As Jeff mentioned, if there is a message that takes too long to process, the group will rebalance and it will affect the whole c

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

We are excited about this release! Excellent work! Thanks Kenny Gorman www.eventador.io > On Feb 22, 2017, at 2:33 AM, Ewen Cheslack-Postava wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.2.0. This is a feature release which includes the complet

Consumer / Streams causes deletes in __consumer_offsets?

Hey users, What causes delete tombstones (value=null) to be sent to the __consumer_offsets topic? I'm observing that a Kafka Streams application that is restarted after a crash appears to be reprocessing messages from the beginning of a topic. I've dumped the __consumer_offsets topic and found th

Re: Consumer / Streams causes deletes in __consumer_offsets?

Hi Mathieu, It could be that the offset retention period has expired. See this: http://stackoverflow.com/questions/39131465/how-does-an-offset-expire-for-an-apache-kafka-consumer-group Th

Re: Heartbeats while consuming a message in kafka-python

Martin, If you called, for example, commitAsync() before you call client.poll() then the commit request will be sent during that period. Guozhang On Wed, Feb 22, 2017 at 2:04 AM, Martin Sucha wrote: > Hi, > > thank you both for responses. > > I forgot to mention which python client I'm using

Re: Consumer / Streams causes deletes in __consumer_offsets?

Hi Eno, Thanks for the quick reply. I think that probably does match the data I'm seeing. This surprises me a bit because my streams app was only offline for a few minutes, but ended up losing its offset. My interpretation is that the source partition had been idle for 24 hours, streams doesn't

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Thanks Ewen :) On Wed, Feb 22, 2017 at 5:15 AM, Kenny Gorman wrote: > We are excited about this release! Excellent work! > > Thanks > Kenny Gorman > www.eventador.io > > > On Feb 22, 2017, at 2:33 AM, Ewen Cheslack-Postava > wrote: > > > > The Apache Kafka community is pleased to announce the r

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Thanks for the release. On Wed, Feb 22, 2017 at 10:32 PM, Becket Qin wrote: > Thanks Ewen :) > > On Wed, Feb 22, 2017 at 5:15 AM, Kenny Gorman wrote: > > > We are excited about this release! Excellent work! > > > > Thanks > > Kenny Gorman > > www.eventador.io > > > > > On Feb 22, 2017, at 2:33

Re: KTable send old values API

Hi Michael, Thanks for reviewing and for the link. As I understand your post, especially > My question is: how much of the TTL idea is about (1) a more granular, per-key expiration of state than what we currently provide, vs. (2) decision-making for whether or not another downstream update shoul

Scaling To Many Kafka Consumers For Particular Topic

Hello, I am currently using a single publisher to publish to a single topic with a single partition. I would like to support many consumers that listen to the same data and cache the topic's data in-memory. What would be the best approach for scaling to potentially hundreds of consumers for this

Re: Scaling To Many Kafka Consumers For Particular Topic

Hi Antony, 1) Kafka only allows one consumer per partition, to guarantee order within partition 2) Only one Kafka broker can be leader for one partition. Based on these 2 building blocks you could split your topic into number of partitions and that way kafka server load gets distributed acr

Re: Scaling To Many Kafka Consumers For Particular Topic

> On Feb 22, 2017, at 1:02 PM, Sunil Patil > wrote: > > Hi Antony, > > > 1) Kafka only allows one consumer per partition, to guarantee order within > partition This is not true. Any number of Consumers can read data from a given Partition. It’s true that within a single Consumer Group, onl

Current offset for partition out of range; reset offset

Hi All, One of the partitions showing the huge lag(21K) and I see the below error in kafkaserver.out log of one of the kafka nodes. Current offset 43294 for partition [PROD_TASK_TOPIC_120,10] out of range; reset offset to 43293 (kafka.server.ReplicaFetcherThread) What is the quick solution,

Re: JMX metrics for replica lag time

Hmm that is a very good question. It seems to me that we did not add the corresponding metrics for it when we changed the mechanism. And your observation is likely to happen, that lag-in-message will not be useful enough to predict / explain why a follower has been kicked out of ISR. Could you fil

Re: Consumer / Streams causes deletes in __consumer_offsets?

Hi Mathieu, In Streams the consumer config "enable.auto.commit" is always forced to false, and a separate "commit.interval.ms" is set. With that even if you do not have any data processed the commit operation will be triggered after that configured period of time. Guozhang On Wed, Feb 22, 2017

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Thanks for driving the release, Ewen. Jun On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.2.0. This is a feature release which includes the completion > of 15 KIPs, over 200 bug fixes and impr

Re: Consumer / Streams causes deletes in __consumer_offsets?

Thanks Guozhang, that clarifies the Streams behavior. I'm imagining that a Streams application might only commit partition offsets that have changed, and therefore a partition that is idle for greater than offsets.retention.minutes might lose its offsets when the app restarts. Does that seem plau

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Awesome thanks a lot! When should we expect the dependencies to be released in Maven? (including 2.12 scala) On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote: Thanks for driving the release, Ewen. Jun On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava wrote: > The Apa

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

I saw them in Maven yesterday? On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek wrote: > Awesome thanks a lot! When should we expect the dependencies to be released > in Maven? (including 2.12 scala) > > On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote: > > Thanks for driving

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11 Am I missing something ? On 23 February 2017 at 9:21:08 am, Gwen Shapira (g...@confluent.io) wrote: I saw them in Maven yesterday? On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek wrote: > Awesome thanks a lot! When should we expec

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

I looked here: https://repository.apache.org/#nexus-search;gav~org.apache.kafka~kafka_2.12kw,versionexpand On Wed, Feb 22, 2017 at 2:26 PM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11 > > Am I missing something ? >

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

The index is stale, the files are there: http://central.maven.org/maven2/org/apache/kafka/kafka_2.11/0.10.2.0/ http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/0.10.2.0/ Ismael On Wed, Feb 22, 2017 at 10:26 PM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > https://mvnrep

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Woohoo! Thanks for running the release, Ewen! -James > On Feb 22, 2017, at 12:33 AM, Ewen Cheslack-Postava wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.2.0. This is a feature release which includes the completion > of 15 KIPs, over 200 bug fix

Question about messages in __consumer_offsets topic

Hi guys, I’m trying to consume from __consumer_offsets topic to get exact committed offset of each consumer. Normally I can see messages like: [eds-els-recopp-jenkins-01-5651,eds-incre-staging-1,0]::[OffsetMetadata[29791925,NO_METADATA],CommitTime 1487090167367,ExpirationTime 1487176567367],

Re: Question about messages in __consumer_offsets topic

__consumer_offsets is a log-compacted topic, and a NULL body indicates a delete tombstone. So it means to delete the entry that matches the key (group, topic, partition tuple). -Todd On Wed, Feb 22, 2017 at 3:50 PM, Jun MA wrote: > Hi guys, > > I’m trying to consume from __consumer_offsets to

Re: hitting the throughput limit on a cluster?

Well, my IOwait sits around 0. Much less than 1% all the time. Any IOwait at all indicates that the application is waiting on the disk, so that’s where your bottleneck will be. As far as open files, it depends on how many log segments and network connections are open. Because I have large clusters

Re: How to stop Kafka Mirror Maker

You should just kill the process without the -9. That will send a SIGHUP. This allows the process to shut down cleanly and flush offset commits before exiting. Otherwise you’re going to have more duplicate messages (or dropped messages, depending on your configuration for offset commits). You shoul

Re: Consumer / Streams causes deletes in __consumer_offsets?

What you said is absolutely right, and sorry I missed that part in the previous email. I think for now it is OK to tune offsets.retention.minutes, as for the long term fix, there are some discussions on this: the retention of offsets today is not tied to whether the group has active members, and i

Re: Question about messages in __consumer_offsets topic

Hi Todd, Thank you so much for your reply. I assume that the broker will produce the tombstone to __consumer_offsets topic when the offset expires? I’m curious how broker notices the offset expires? Does it store the offset message in memory and periodically check if someone expires? Thanks, J

Re: Question about messages in __consumer_offsets topic

The __consumer_offsets topic should also get a tombstone message as soon as a topic is deleted. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Wed, Feb 22, 2017 at 5:59 PM, Jun MA wrote: > Hi Todd, > > Thank you so much for you

Simple data-driven app design using Kafka

Hello Kafka Users, I started using Kafka a couple of weeks ago an am very impressed! I've gotten the hang of producing, and now it's time for consuming. My applications (Scala) don't work quite like the examples, but I think it's a pretty basic architecture: - Suppose you have a several top

Re: Simple data-driven app design using Kafka

Pete, I think this excellent post covers what you are looking for: https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/ --Vahid From: Peter Figliozzi To: users@kafka.apache.org Date: 02/22/2017 07:29 PM Subject:Simple data-driv

Re: Question about messages in __consumer_offsets topic

Hi Jun, I ran into the same question today (see thread, subject: Consumer / Streams causes deletes in __consumer_offsets?), and here's what Eno and Guozhang helped me understand: There are broker-level configuration values called "offsets.retention.minutes" and "offsets.retention.check.interval.m

Re: JMX metrics for replica lag time

Just wondering, for what particular Kafka version is this applicable? On Thu, Feb 23, 2017 at 2:38 AM, Guozhang Wang wrote: > Hmm that is a very good question. It seems to me that we did not add the > corresponding metrics for it when we changed the mechanism. And your > observation is likely to

Re: Consumer / Streams causes deletes in __consumer_offsets?

Hi Guozhang, On Thu, Feb 23, 2017 at 2:48 AM, Guozhang Wang wrote: > With that even if you do > not have any data processed the commit operation will be triggered after > that configured period of time. > The above statement is confusing. As per this thread , offsets are o

Re: Consumer / Streams causes deletes in __consumer_offsets?

Mahendra, That is right, what I meant is that at the end of each loop in the thread, it will check against the commit internal and see if it should do so. That means, commit will only happen after any records have been completely processed in the topology, and that also means that the actual commi

Re: Consumer / Streams causes deletes in __consumer_offsets?

Thanks! On Thu, Feb 23, 2017 at 10:49 AM, Guozhang Wang wrote: > Mahendra, > > That is right, what I meant is that at the end of each loop in the thread, > it will check against the commit internal and see if it should do so. That > means, commit will only happen after any records have been comp