Is there any way to find out whether "kafka" is used as offset storage or "zookeeper"

Hi, I have set offset.storage=kafka and dual.commit.enabled=false in the consumer properties and restarted the brokers. I can send and receive messages from Kafka. I just want to make sure that "kafka" is used as offset storage not "zookeeper". So Is there any way to see whether "kafka" is used

Slow ISR catch-up

Hi Folks, Request your expertise on my doubt here. *My setup:-* 5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB) Using Kafka 0.8.2.1 with modified ProducerPerformance.scala I've modified ProducerPerformance.scala to send custom ASCII data, instead of Byte Array of Zeroes *server.prope

Re: How to monitor lag when "kafka" is used as offset.storage?

You can use the emailer config in Burrow to send alerts directly (it will monitor specific groups and send emails out when there is a problem). If you need something more complex than that, I think the best practice is always to send the output into an general alert/notification system. -Todd On

API to query cluster metadata on-demand

Is there a basic interface in the new client APIs to get the list of topics on a cluster, and get information on the topics (offsets, sizes, etc), without having to deal with a producer or consumer? I just want a basic synchronous API to query the metadata as-is. Does this exist in some form? T

[VOTE] 0.8.2.2 Candidate 1

This is the first candidate for release of Apache Kafka 0.8.2.2. This only fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in 0.8.2.1. Release Notes for the 0.8.2.2 release https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/RELEASE_NOTES.html *** Please download, t

Competing customers

I imagine this has been asked before, but I have googled around quite a bit and can't really find a clear answer. Apologies in advance, though .. I'm interested in Kafka setups that allow for competing customers. I'll have one topic where a lot of messages get published to, and I'd like to be ab

Re: Competing customers

Yeah, scaling through adding partitions ("sharding") is a basic feature of Kafka. We expect topics to have many partitions (at least as many as number of consumers), and each consumer to get a subset of the messages by getting a subset of partitions. This design gives Kafka its two biggest advanta

RE: Competing customers

Great, thanks - that does help. I'll kick off some partitions, then. :) (I think I saw your video lectures on safaribooksonline! I should probably have paid better attention..) Joris Peeters Software Developer Research and Data Technology T: +44 (0) 20 8576 5800 -Original Message- Fro

Re: API to query cluster metadata on-demand

Ah, I wish. We are working on it :) On Thu, Sep 3, 2015 at 9:10 AM, Simon Cooper < simon.coo...@featurespace.co.uk> wrote: > Is there a basic interface in the new client APIs to get the list of > topics on a cluster, and get information on the topics (offsets, sizes, > etc), without having to de

Re: Slow ISR catch-up

The test uses the old producer (we should fix that), and since you don't specify --sync, it runs async. The old async producer simply sends data and doesn't wait for acks, so it is possible that the messages were never acked... On Thu, Sep 3, 2015 at 7:56 AM, Prabhjot Bharaj wrote: > Hi Folks, >

Re: API to query cluster metadata on-demand

If you don’t mind doing it with a C CLI: https://github.com/edenhill/kafkacat $ kafkacat -L -b mybroker But, uhhh, you probably want a something in the Java API. :) > On Sep 3, 2015, at 13:58, Gwen Shapira wrote: > > Ah, I wish. > > We are working on it :) > > On Thu, Sep 3, 2015 at 9:10

Re: API to query cluster metadata on-demand

What Gwen said :) We developed a python web service internally called Menagerie that provides this functionality for both Kafka and Zookeeper. We use it to drive a web dashboard for stats, our (old style) lag checking, and some other CLI tools. Unfortunately it ties into too much internal LinkedIn

Re: Slow ISR catch-up

Hi, Can I use sync for acks = -1? Regards, Prabhjot On Sep 3, 2015 11:49 PM, "Gwen Shapira" wrote: > The test uses the old producer (we should fix that), and since you don't > specify --sync, it runs async. > The old async producer simply sends data and doesn't wait for acks, so it > is possibl

Re: Slow ISR catch-up

Yes, this should work. Expect lower throughput though. On Thu, Sep 3, 2015 at 12:52 PM, Prabhjot Bharaj wrote: > Hi, > > Can I use sync for acks = -1? > > Regards, > Prabhjot > On Sep 3, 2015 11:49 PM, "Gwen Shapira" wrote: > > > The test uses the old producer (we should fix that), and since yo

Huge Kafka Message size ( 386073344 ) in response

Hi all, I'm using the Kafka.Net library for implementing the Kafka Producer. One issue I find out is that sometimes it reads the response from kafka server, which indicates a huge message size 386073344. Apparently something must be wrong. But I'm not sure if it's a special flag that Kafka.net

latency test

I am running a producer latency test. When using 92 producers in 92 physical node publishing to 4 brokers, the latency is slightly lower than using 8 brokers, I am using 8 partitions for the topic. I have rerun the test and it gives me the same result, the 4 brokers scenario still has lower latenc