Apache Kafka Topic Design

Dear Kafka Users, I have been experimenting with Kafka and I'm currently trying to come up with a design for the set of topics that we would need for our use case. We are reading a set of signals from a device and we would be having a set of those device. Say, we have device1, device2 and so on w

Re: Kafka Consumer Example source code seems not working correctly

Hello. I've been trying to debug the source code and I noticed that the execution of source code was stopped at the lane "ConsumerRecords records = consumer.poll(100);". Furthermore, I also tested ProducerExample, but it also seems not working correctly, at least the source code generated messages

Kafka Consumer Example source code seems not working correctly

Hello. I'm a new who are getting started to learn Kafka. I'm now trying to develop a sample java source code to test the new Kafka Consumer, but it seems not working correctly. I've been testing both of Kafka Consumer versions; the old one called as High Level Consumer and another one called as a

Re: Kafka + ZooKeeper on the same hardware?

> On Jan 18, 2016, at 12:21 PM, Dick Davies wrote: > > Started an Ansible playbook using the Confluent platform RPM distro, > and it seems that co-locates zookeepers > on the brokers. > > So I'm assuming it's fine (at least on 0.9.x for the reasons Todd mentioned). > > Does anyone know if the Con

Re: Kafka + ZooKeeper on the same hardware?

Started an Ansible playbook using the Confluent platform RPM distro, and it seems that co-locates zookeepers on the brokers. So I'm assuming it's fine (at least on 0.9.x for the reasons Todd mentioned). Does anyone know if the Confluent distro is supposed to be production-ready? I switched our te

Re: Possible WAN Replication Setup

Hey Ben, Thank you for the heads up. I must have misunderstood the docs...my reading was that a commit doesn't complete until all the in-sync replicas (in this case, 1) have acknowledged committing to the log. Would I be incorrect in extrapolating that if an entire data center lost power, a 3+ nod

Re: Possible Error Code for Kafka API Calls

That would be even better Gwen. Ismael I'm wondering if the protocol docs can be auto-generated from our code to a large extent. Or if we can enhance our protocol definition classes a bit to make them self-documenting (the way we did for configuration). Regarding Dana's suggestion: I think you ne

Re: Possible Error Code for Kafka API Calls

Awesome! Fwiw, my understanding of the possible produce errors is: LeaderNotAvailable NotLeaderForPartition UnknownTopicOrPartition InvalidMessage MessageSizeTooLarge RecordListTooLarge InvalidTopic NotEnoughReplicas NotEnoughReplicasAfterAppend InvalidRequiredAcks TopicAuthorizationFailed The o

Re: Possible Error Code for Kafka API Calls

I added what I found in the code comments to the wiki. Note that there are some gaps. For example if anyone can fill in the producer error codes, it will be awesome :) On Mon, Jan 18, 2016 at 9:17 AM, Gwen Shapira wrote: > I'm wondering if the protocol docs can be auto-generated from our code to

Re: Possible Error Code for Kafka API Calls

I'm wondering if the protocol docs can be auto-generated from our code to a large extent. Or if we can enhance our protocol definition classes a bit to make them self-documenting (the way we did for configuration). Regarding Dana's suggestion: I think you need special wiki-edit privileges. If you

Re: Possible Error Code for Kafka API Calls

In addition to this, it would be great to move the protocol docs to the docs folder of the git repo: https://github.com/apache/kafka/tree/trunk/docs This way, we can ensure that the protocol docs are updated at the same time as the protocol code. Ismael On Mon, Jan 18, 2016 at 3:36 PM, Dana Pow

Re: Possible Error Code for Kafka API Calls

Not exactly - there is some documentation in the source code, but I agree that a wiki on this would be extremely useful. Can anyone create a wiki page? If so, I'm happy to get something started. It is really the missing piece for folks writing custom clients / anything at the api layer. -Dana On

Possible Error Code for Kafka API Calls

dear all, I found a guide in Kafka protocol (Kafka Protocol Guide ), the guide describes API calls complete with their request and response format, and list of error codes with t

Re: Producer Config Error - Kafka 0.9.0.0

Hi Joe, bootstrap.servers is the configuration for new *java* producer. Cheers, Steve On Mon, Jan 18, 2016, 5:41 PM Joe San wrote: > Dear Kafka Users, > > Is there a bug with the producer config? > > I have asked a question on Stackoverflow: > > > http://stackoverflow.com/questions/34851412/ap

Producer Config Error - Kafka 0.9.0.0

Dear Kafka Users, Is there a bug with the producer config? I have asked a question on Stackoverflow: http://stackoverflow.com/questions/34851412/apache-kafka-producer-config-error As per the documentation, I need to only provide bootstrap.servers but, when I run my producer client, I get a mes