Using topic or partition for large size of consumers (subscribers)?

2016-10-07 Thread jupiter
Hi, I can run a producer to send data to a particular partition within a topic by specifying the topic ID and the partition ID, then a consumer can receive a particular partition within a topic by subscribing with the topic ID and partition ID. If I have n consumers, each subscribes to a topic/pa

topic / partition granularity

2016-10-15 Thread jupiter
Hi, Obvious, it is not practical and not scalable to use one partition links one account., we are going to use partition sharing scheme where one topic partition will be shared by a group of accounts, which means in account consumer, we have to check in user message level (not kafka message level)

ZK WARN Cannot open channel to 3 at election address .... java.net.ConnectException: Connection refused

2016-10-19 Thread jupiter
Hi, I am running ZK from version 0.10.0.1 on 3 machines, the hostnames haven't been registered on DNS so all defined in /etc/hosts, there are no firewall between them. I saw many posts online pointed to some bugs in the zookeepers and there are many patches to fix it. Has the zookeeper in 0.10.0.1