Re: ISR differs between Kafka Metadata and Zookeeper

I am seeing this behavior using librdkafka, as is another user. Listing the topic metadata with the tool provided with Kafka (kafka-topic.sh) shows all replicas in the ISR. However, using kafkacat[1] (built with librdkafka) shows that many ISRs are missing some replicas. I talked with Magnus (au

Re: Disactivating Yammer Metrics Monitoring

Hi Daniel, Thank you for your answer. It's possible that I didn't understood something, if so correct me please. >From what I understood, from the kafka doc #monitoring , kafka use Yammer Metrics for monitoring the servers (the brokers) and

subscribe kafka mail list

Hi, I am interested in Kafka, I want to subscribe kafka mail lists, Thanks.

Re: subscribe kafka mail list

http://kafka.apache.org/contact.html François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communication - CS Games 2014 Jeux de Génie 2011 à 2014 Mag

Copying messages from a single partition topic to a multi-partition topic

Hi, We have an interesting problem to solve due to a very large traffic volumes on particular topics. In our initial system configuration we had only one partition per topic, and in in a couple of topics we have built up huge backlogs of several million messages that our consumers are slowly proce

Re: Copying messages from a single partition topic to a multi-partition topic

I would look at writing a service that reads from your existing topic and writes to a new topic with (e.g. four) partitions. You will also need to pay attention to the partitioning policy (or implement your own), as the default hashing in the current kafka version default can lead to poor distr

Re: Copying messages from a single partition topic to a multi-partition topic

Agree. Why not write a service that reads from existing topics and writes to new ones. Point existing producers to the new partitions wait for the service to finish reading. Since you have only one partition right now, I'd assume you don't have any partitioning logic per se. Think about that since

Re: ISR differs between Kafka Metadata and Zookeeper

This may be due to https://issues.apache.org/jira/browse/KAFKA-1367 On Thu, Oct 03, 2013 at 04:17:45PM -0400, Florian Weingarten wrote: > Hi list, > > I am trying to debug a strange issue we are seeing. We are using > "Sarama" [1], our own Go implementation of the Kafka API. > > Somehow, we eith

Re: How to use kafka as flume source.

Just to update (better late than never!): The Kafka source & sink for Flume were updated to latest Kafka version and improved a bit (offsets are now committed after data is written to Flume channel). If you build Flume from trunk, you'll get these. Gwen On Sun, Aug 3, 2014 at 10:31 AM, Andrew Ehr

How to use RPC mechanism in Kafka?

Hi, Please tell me how to use request/response mechanism in kafka? Thanks Lavish Goel

Re: Handling errors in the new (0.8.2) Java Client's Producer

Hello Andrew, I think you would want a sync producer for your use case? You can try to call get() on the returned metadata future of the send() call instead of using a callback; the pattern is something like: for (message in messages) producer.send(message).get() The get() call will block un

Re: How to use RPC mechanism in Kafka?

Do you mean that you want to know the protocol? https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Guozhang On Fri, Sep 19, 2014 at 12:18 PM, lavish goel wrote: > Hi, > > Please tell me how to use request/response mechanism in kafka? > Thanks > Lavish Goel > --

Re: Handling errors in the new (0.8.2) Java Client's Producer

Hi Guozhang, I specifically to not want to do a get() on every future. This is equivalent to going "sync"ed, which is a performance killer for us. What I am looking for is some way to tell a producer to forget about queued messages. I have done some more testing on my solution and it needs some w

Re: Handling errors in the new (0.8.2) Java Client's Producer

Andrew If you see BaseProducer.scala, there's the following code snippet override def send(topic: String, key: Array[Byte], value: Array[Byte]) { val record = new ProducerRecord(topic, key, value) if(sync) { this.producer.send(record).get() } else { this.producer.send(reco

Re: Disactivating Yammer Metrics Monitoring

Hi, I don't have any source or configs handy to check things, but you are saying you've configured Kafka to use GraphiteReporter, right? So why not remove that config, so metrics stop being sent to Graphite if your Graphite setup is suffering? If you do that and you still want to see your Kafka

Re: Interaction of retention settings for broker and topic plus partitions

That's right. The rule is that a log segment is deleted if either the size or the time limit is reached. Log sizes are per partition. Thanks, Jun On Thu, Sep 18, 2014 at 2:55 PM, Cory Watson wrote: > Hello all! > > I'm curious about the interaction of server and topic level retention > setting

Re: How to use RPC mechanism in Kafka?

Thank you for your response. I want to implement request/response type model. For eg. I have a producer that publish a message(He wants to know some status) to topic. A consumer pulls the message and processes the request. Now I want that, the response of this request should go that producer. Can y