I have a problem about kafka (0.7.2)

Hello kafka : I find a problem about kafka , I build a kafka cluster with 3 zookeepers and 7 kafka brokers, it works well when developing ,but when I deploy it online ,I find that :each kafka producer keeps a lot of connections with zookeeper . On a kafka producer machine ,I use ` ss –anop | gr

Re: [DISCUSSION] adding the serializer api back to the new java producer

Jun, Thanks for summarizing this - it helps confirm for me that I did not misunderstand anything in this thread so far; and that I disagree with the premise that the steps in using the current byte-oriented API is cumbersome or inflexible. It involves instantiating the K-V serializers in code (as

Re: metrics about how behind a replica is?

replica.lag.max.messages only controls when a replica should be dropped out of the in-sync replica set (ISR). For a message to be considered committed, it has to be added to every replica in ISR. When the producer uses ack=-1, the broker waits until the produced message is committed before acknowle

Re: consumer groups

" If all the consumers stop listening how long will Kafka continue to store messages for that group?" Kafka retains data for set amount of time, regardless of whether anyone is listening or not. This amount of time is configurable. Because Kafka performance is generally constant with the amount of

RE: Built-in replication for broker

Thanks for reply! > Date: Tue, 16 Dec 2014 18:23:39 -0800 > Subject: Re: Built-in replication for broker > From: n...@confluent.io > To: users@kafka.apache.org > > Replication is available in Kafka 0.8.0 onwards. > > On Tue, Dec 16, 2014 at 4:31 PM, Haoming Zhang > wrote: > > > > Dear developer

Re: Built-in replication for broker

Replication is available in Kafka 0.8.0 onwards. On Tue, Dec 16, 2014 at 4:31 PM, Haoming Zhang wrote: > > Dear developers, > > In the paper "Kafka: A Distributed Messaging System for Log Processing, > Jay Kreps, Neha Narkhede, Jun Rao from LinkedIn, at NetDB workshop 2011", > > you mentioned thi

Built-in replication for broker

Dear developers, In the paper "Kafka: A Distributed Messaging System for Log Processing, Jay Kreps, Neha Narkhede, Jun Rao from LinkedIn, at NetDB workshop 2011", you mentioned this: "If a broker goes down, any message stored on it not yet consumed becomes unavailable. If the storage system on

consumer groups

Hi, I am planning to use kafka for a work queue type use case with multiple consumers. I also plan to use it with multiple consumer groups. What is not clear to me from the documentation is how to define+manage consumer groups or if that is possible at all. It appears to me that a consumer group i

Re: [DISCUSSION] adding the serializer api back to the new java producer

Joel, With a byte array interface, of course there is nothing that one can't do. However, the real question is that whether we want to encourage people to use it this way or not. Being able to flow just bytes is definitely easier to get started. That's why many early adopters choose to do it that

Re: How do I create a consumer group

>Yes - see the offsets.topic.num.partitions and >offsets.topic.replication.factor broker configs. Joel, that exactly what I was looking for. I'll look into that and the source for OffsetsMessageFormatter later today! thanks svante >

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

Any suggestions what might be going on here. We are very much blinded here and our application is getting effected due to this. -Mohit On Tue, Dec 9, 2014 at 8:41 PM, Mohit Kathuria wrote: > > Neha, > > The same issue reoccured with just 2 consumer processes. The exception was > related to conf