Re: Slides and Video from Kafka Summit are now available

2016-05-13 Thread Ben Davison
Hi Gwen, Awesome stuff, can't wait to go through these -- I noticed that one link didn't go anywhere. Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData Will this be posted anytime? Thanks, Ben On Fri, May 13, 2016 at 8:05 PM, Gwen Shapira wrote: > Hey Kafka Community,

Suggestions with hostnames in a Kafka cluster

2016-05-13 Thread R Krishna
As a newbie, just setup my first Kafka 3 node, each in its own host cluster and its own ZK. Everything went fine, I could see three brokers registered in all three ZK /brokers/ids until I created a topic with this exception: ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replica

ISR shrinking/expanding problem

2016-05-13 Thread Russ Lavoie
Hello, I moved an entire topic from one set of brokers to another set of brokers. The network throughput was so high, that they fell behind the leaders and dropped out of the ISR set. How can I recover from this? Thanks!

Re: Connect MemoryOffsetBackingStore creates a non-daemon thread, prevents clean shutdown

2016-05-13 Thread Peter Davis
Created https://issues.apache.org/jira/browse/KAFKA-3710 Regards, Peter Davis (206) 713-7352 davi...@gmail.com On Fri, May 13, 2016 at 7:02 AM, Peter Davis wrote: > I tried. JIRA is locked down. > > -Peter > > > > On May 12, 2016, at 18:14, Liquan Pei wrote: > > > > Hi Peter, > > > > That s

Slides and Video from Kafka Summit are now available

2016-05-13 Thread Gwen Shapira
Hey Kafka Community, It was great seeing so many of you at the Kafka summit last month. Hope you had fun and learned a lot. I certainly did. Looking forward to meet all of you again at the next summit :) For those of you who missed the event, or those who attended but are sorry they couldn't see

Re: compacted log TTLs

2016-05-13 Thread Guozhang Wang
Peter, Wesley, thanks for your use cases. There is a KIP discussion about adding a timestamp based log deletion policy into Kafka along side with compaction; and I'm thinking whether it makes sense to enable both log deletion and log compaction for the general cases of changelog data with expirati

Re: Kafka Consumer rebalancing frequently

2016-05-13 Thread Cees de Groot
What Abhinav said. To give some context: the common cause of frequent rebalances is that your consumer takes too long to process batches. As long as you don't call into the consumer library, heartbeats aren't sent so if you take too long working through a batch, the broker things your consumer is g

Re: Kafka Consumer rebalancing frequently

2016-05-13 Thread Abhinav Solan
Hi Sahitya, Try reducing max.partition.fetch.bytes in your consumer. Then also increase heartbeat.interval.ms, this might help in to delay the consumer rebalance of your inbound process is taking more time than this - Abhinav On Fri, May 13, 2016 at 5:42 AM sahitya agrawal wrote: > Hi, > > I a

Re: compacted log TTLs

2016-05-13 Thread Wesley Chow
Yes, also classic caching, where you might use memcache with TTLs. But a different use case for us is sessionizing. We push a high rate of updates coming from a browser session to our Kafka cluster. If we don’t see an update for a particular session after some period of time, we say that session

Re: Connect MemoryOffsetBackingStore creates a non-daemon thread, prevents clean shutdown

2016-05-13 Thread Peter Davis
I tried. JIRA is locked down. -Peter > On May 12, 2016, at 18:14, Liquan Pei wrote: > > Hi Peter, > > That seems to be an issue. Do you want to file a JIRA on this? > > Thanks, > Liquan > >> On Thu, May 12, 2016 at 5:01 PM, Peter Davis wrote: >> >> I'm having an issue running a Connect s

Kafka Consumer rebalancing frequently

2016-05-13 Thread sahitya agrawal
Hi, I am using new Kafka consumer API ( 0.9.0.0) . I created 100 partitions of a topic and started only one consumer to consume. Many of times , In consumer logs I see lot of rebalancing activity and no object is consumed due to that. Is this a known issue? Please let me know if some body can he