Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

2013-10-05 Thread Guozhang Wang
Hello Jacky, Have you read this FAQ: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F Guozhang On Fri, Oct 4, 2013 at 10:41 PM, Jiang Jacky wrote: > It is very weird, I have a kafka cluster in EC2, There is no any problem

Re: testing issue with reliable sending

2013-10-05 Thread Neha Narkhede
Shouldn't this be part of the contract? It should be able to make sure this happens before shutting down, no? The leader writes messages to its local log and then the replicas consume messages from the leader and write those to their local logs. If you set request.required.acks=1, the ack is sent

Re: testing issue with reliable sending

2013-10-05 Thread Jason Rosenberg
Thanks for the explanation Neha.still holding out hope. So, if request.required.acks=-1, how does the leader confirm that the other brokers have consumed the message, before acking to the producer? Does the leader just wait for the followers in the ISR to consume? Or does the leader have

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

2013-10-05 Thread Jiang Jacky
Hi, I tried to setup the host.name in servier.properties, it doesn't work. I believe it is the network security issue. However, I create a new instance in the same security group without kafka, zookeeper, it does work, it can still produce to kafka server. but when I change to another ec2 account,

Managing Millions of Paritions in Kafka

2013-10-05 Thread Ravindranath Akila
Initially, I thought dynamic topic creation can be used to maintain per user data on Kafka. The I read that partitions can and should be used for this instead. If a partition is to be used to map a user, can there be a million, or even billion partitions in a cluster? How does one go about designi