Re: Partition election on consumer

Thanks Neha, really appreciate your assistance 2013/10/9 Neha Narkhede > Kafka's consumer rebalancing strategy is explained in detail here - > http://kafka.apache.org/documentation.html#distributionimpl > Hope that helps! > > -Neha > > > On Tue, Oct 8, 2013 at 11:42 PM, Markus Roder >wrote: >

Re: Best way to start / stop / restart a Consumer?

Thanks Jun, Jira issue has been filled: https://issues.apache.org/jira/browse/KAFKA-1083 By the way, what is the recommended way to start, stop and restart a ConsumerConnector in the same running JMV? Thanks, 2013/10/10 Jun Rao > Each time we create a new consumer connector, we assign a rand

Re: Best way to start / stop / restart a Consumer?

Best way to shutdown is to invoke the shutdown() API. To restart, you need to createMessageStreams() Thanks, Neha On Oct 11, 2013 6:10 AM, "Tanguy tlrx" wrote: > Thanks Jun, > > Jira issue has been filled: > https://issues.apache.org/jira/browse/KAFKA-1083 > > By the way, what is the recommended

Re: Best way to start / stop / restart a Consumer?

Hi Neha, Thanks for the tip but calling again createMessageStreams() on a ConsumerConnector object that has been shut down throws a NullPointerException. Did you manage to get it work? 2013/10/11 Neha Narkhede > Best way to shutdown is to invoke the shutdown() API. To restart, you need > to

Re: Kafka cannot produce message

Apache email doesn't allow attachment. Could you try the latest code in the 0.8 branch and see if the same issue exists? Thanks, Jun On Thu, Oct 10, 2013 at 10:20 PM, Jiang Jacky wrote: > > for those 2 logs directly in my ~/kafka_2.8.0-0.8.0-beta1, Maybe I did not > set the log output to indi

Re: Is there a way to pull out kafka metadata from zookeeper?

You could use a broker list or a VIP to deal with broker failures. Thanks, Jun On Thu, Oct 10, 2013 at 10:50 PM, hsy...@gmail.com wrote: > Hi guys, > > I'm trying to maintain a bunch of simple kafka consumer to consume messages > from brokers. I know there is a way to send TopicMetadataReques

Re: Question about auto-rebalancing

If you are using simple consumer, you are responsible for dealing with leader replica changes. When the leader changes, an error code will be returned in the fetch response and you need to refresh metadata and retry the fetch request to the new leader. For details, see https://cwiki.apache.org/conf

Re: Messages from producer are immediately going to /tmp/logs in kafka

Those messages could still be in file system pagecache and may not be flushed to disks. Thanks, Jun On Thu, Oct 10, 2013 at 11:20 PM, Monika Garg wrote: > Hi, > > In kafka-0.8 there are three important properties given for > > log.flush.interval.messages=1 > > log.flush.interval.ms=90

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

The issue doesn't seem to point to anything. Do you mind sending it in a pastebin? On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky wrote: > Hi, Thanks Joe > I have disabled the firewall, but I did not tried that again, I changed > another way by using Netty sending the socket package to kafka ser

Re: kafka.common.UnknownTopicOrPartitionException

The state-change.log captures critical state change information about all partitions in a Kafka cluster and we have toolsthat you can use on the state-change.log to self troubleshoot any

Re: Kafka cannot produce message

Does the change to /etc/hosts resolve this issue as well? On Fri, Oct 11, 2013 at 8:16 AM, Jun Rao wrote: > Apache email doesn't allow attachment. Could you try the latest code in the > 0.8 branch and see if the same issue exists? > > Thanks, > > Jun > > > On Thu, Oct 10, 2013 at 10:20 PM, Jian

Re: Best way to start / stop / restart a Consumer?

Can you send around your code snippet and related NPE stack trace? On Fri, Oct 11, 2013 at 8:01 AM, Tanguy tlrx wrote: > Hi Neha, > > Thanks for the tip but calling again createMessageStreams() on a > ConsumerConnector object that has been shut down throws a > NullPointerException. > > Did you

Re: Is there a way to pull out kafka metadata from zookeeper?

If, for some reason, you don't have access to a virtual IP or load balancer, you need to round robin once through all the brokers before failing a TopicMetadataRequest. So unless all the brokers in your cluster are down, this should not be a problem. Thanks, Neha On Thu, Oct 10, 2013 at 10:50 PM

Re: Anyone running kafka with a single broker in production? what about only 8GB ram?

Hello, In most cases of Kafka, network bottleneck will be hit before the disk bottleneck. So maybe you want to check your network capacity to see if it has been saturated. Guozhang On Thu, Oct 10, 2013 at 3:57 PM, Bruno D. Rodrigues < bruno.rodrig...@litux.org> wrote: > A 10/10/2013, às 23:14,

Re: Anyone running kafka with a single broker in production? what about only 8GB ram?

> On Thu, Oct 10, 2013 at 3:57 PM, Bruno D. Rodrigues < > bruno.rodrig...@litux.org> wrote: > >> My personal newbie experience, which is surely completely wrong and >> miss-configured, got me up to 70MB/sec, either with controlled 1K messages >> (hence 70Kmsg/sec) as well as with more random data

Re: Anyone running kafka with a single broker in production? what about only 8GB ram?

Make sure the fetch batch size and the local consumer queue sizes are large enough, setting them too low will limit your throughput to the broker<->client latency. This would be controlled using the following properties: - fetch.message.max.bytes - queued.max.message.chunks On the producer side y

Re: Question about auto-rebalancing

Hi Jun, Thanks for your reply, but in a real cluster, one broker could serve different topics and different partitions, the simple consumer only has knowledge of brokers that are available but it has no knowledge to decide which broker is best to pick up to consume messages. If you don't choose c

Best way to start / stop / restart a Consumer?

Re: starting/stopping - right now you would have to call shutdown and create a new consumer connector since you can call createMessageStreams only once per connector. On Friday, October 11, 2013, Tanguy tlrx wrote: > Thanks Jun, > > Jira issue has been filled: > https://issues.apache.org/jira/bro

Re: Anyone running kafka with a single broker in production? what about only 8GB ram?

Producer: props.put("batch.num.messages", "1000"); // 200 props.put("queue.buffering.max.messages", "2"); // 1 props.put("request.required.acks", "0"); props.put("producer.type", "async"); // sync // return ++this.count % a_numPartitions; // just

Re: Kafka startup, no broker partitions consumed by consumer thread

Jun, Thank you! That helped. Thanks, Viktor

Re: Is there a way to pull out kafka metadata from zookeeper?

Thanks guys! But I feel weird. Assume I have 20 brokers for 10 different topics with 2 partitions and 2 replicas for each. For each consumer consumes different topic/replica I have to specify those 20 brokers and go over all of them to know which broker is alive. And even worse how about I dynamic

Re: Anyone running kafka with a single broker in production? what about only 8GB ram?

I'm also curious to know what is the limiting factor of kafka write throughput? I've never seen reports higher than 100mb/sec, obviously disks can provide much more. In my own test with single broker, single partition, single replica: bin/kafka-producer-perf-test.sh --topics perf --threads 10 --br

Re: Is there a way to pull out kafka metadata from zookeeper?

>> For each consumer consumes different topic/replica I have to specify those 20 brokers and go over all of them to know which broker is alive. And even worse how about I dynamically add new broker into the cluster and remove the old one TopicMetadataRequest is a batch API and you can get metadata

Re: Is there a way to pull out kafka metadata from zookeeper?

What if the broker it tries to get the TopicMetadata from is down? I assume a broker is more like to be down than a zookeeper. Thanks, Siyuan On Fri, Oct 11, 2013 at 2:39 PM, Neha Narkhede wrote: > >> For each consumer consumes different > topic/replica I have to specify those 20 brokers and go

Re: Is there a way to pull out kafka metadata from zookeeper?

Why not ask zookeeper for the list of brokers and then ask a random broker for the metadata (and repeat if the broker is down), even if it's two calls. Heck it already does unnecessary connections. It connects to a broker, gets the metadata, disconnects, and then connects again for the data. If it

Re: Is there a way to pull out kafka metadata from zookeeper?

That's why I'm asking, I would like to see a kafka zookeeper client api to get TopicMetadata instead of my own hacky way to query the zookeeper Thanks! Best, Siyuan On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues < bruno.rodrig...@litux.org> wrote: > Why not ask zookeeper for the list of br

Re: Question about auto-rebalancing

Hello Siyuan, For the automatic leader re-election, yes we are considering to make it work. Could you file a JIRA for this issue? For the high-level consumer's rebalancing logic, you can find it at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredicttheresultsoftheconsumerrebaba

Re: Question about auto-rebalancing

We already have a JIRA for auto rebalance. I would be working on this soon. KAFKA-930 On 10/11/13 5:39 PM, "Guozhang Wang" wrote: >Hello Siyuan, > >For the automatic leader re-election, yes we are considering to make it >work. Could you file a

Re: Is there a way to pull out kafka metadata from zookeeper?

Hello Siyuan, Pre 0.8 Kafka does have a zookeeper based producer api, and this has been removed in 0.8. You can find some of the reasons in this JIRA: https://issues.apache.org/jira/browse/KAFKA-369 Guozhang On Fri, Oct 11, 2013 at 5:16 PM, hsy...@gmail.com wrote: > That's why I'm asking, I

Re: Question about auto-rebalancing

Oh, Sriram, Thank you very much! On Fri, Oct 11, 2013 at 5:44 PM, Sriram Subramanian < srsubraman...@linkedin.com> wrote: > We already have a JIRA for auto rebalance. I would be working on this soon. > > KAFKA-930 > > > > On 10/11/13 5:39 PM, "G