If I understand your comment correctly you are trying to use kafka topics as
per-endpoint message queues.
I may be mistaken, but to me Kafka seems does not really seem like a good match
for that. For such a purpose you will eventually want something that is not
actually a queue - a means to
Thanks for the replies. I don't think Kafka quite fits our use case,
unfortunately. To abstractly answer Edward's question: in a system with
lots of users, we were considering having a topic per user (such that an
individual user can connect from a number of endpoints and receive events,
including
Zookeeper will not be the only problem. The first is that each topic is a
directory on the file system. Each of those is going to have files inside
it. This is going to be fairly overwhelming for the file system. Also I can
not speak for the internals but there may be cases where this many topics
a
A node being added will generally not lead to any leader election. A node
being removed will lead to leader election. You can force leader election
using the preferred replica election command -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaL
I didn't see any auto leader election for adding nodes. The data are still
skewed on the old nodes. You have to force it by running script?
On Wed, Nov 13, 2013 at 6:41 AM, Neha Narkhede wrote:
> At those many topics, zookeeper will be the main bottleneck. Leader
> election process will take ver
At those many topics, zookeeper will be the main bottleneck. Leader
election process will take very long increasing the unavailability window
of the cluster.
Thanks,
Neha
On Nov 13, 2013 4:49 AM, "Joe Freeman" wrote:
> Would I be correct in assuming that a Kafka cluster won't scale well to
> sup
