Right, in 0.7, partitioner is only supported for zookeeper based producer. Thanks,
Jun On Wed, Dec 12, 2012 at 4:44 PM, Subhash Agrawal <sagra...@actuate.com>wrote: > Hi, > > I have one more question related to this setup. If we don't use zookeeper > in single Kafka broker setup, what will be the loss in > terms of functionality? I noticed in ProducerConfig.scala file, that if > brokerlist is configured, then it ignore partitioner class > configuration. > > Thanks > Subhash Agrawal > > > -----Original Message----- > From: Jamie Wang [mailto:jamie.w...@actuate.com] > Sent: Wednesday, December 12, 2012 2:26 PM > To: users@kafka.apache.org > Subject: Design questions and your opinion and suggestions are appreciated > > Hi, > > We are incorporating Kafka as part of our centralized log aggregation > service for a clustered server system. Our current thinking on the design > as follows: Each of our clustered server will produce 8 different logs. A > typical cluster will have about 4 or 5 nodes right now. Our design is to > create a topic per each type of log and the number of partitions within > each topic will equal to the number of nodes in the cluster. Therefore > with a cluster of 5 nodes, we will have 8 topics and 5 partitions within > each topic. > > On each of the node, we have a producer that has 8 threads. Each thread > will always write to the same topic and partition. > > On the consumer side, we plan to have an aggregation client that has 8 > threads. Each thread will pull from the same topic using round robin > fashion to cycle through each partition within the topic. > > Couple of design questions we would like to hear your opinion and any > suggestions are appreciated! > > 1) If we dynamically adds another node into the cluster, base on our > design above, we would want to create a new partition and add to each of > the 8 topics. Is this doable and how do I do that? > > 2) Can a producer send message to a specific partition? Base on our > limited understanding so far, in a clustered kafka system, a producer sends > message to the cluster, how does the messages being distributed among > different brokers/partitions? > > 3) In our system, we also have a c++ process manager that's monitoring > over kafka broker process. is there a heartbeat in the broker that we can > ping? Or where can I find the C++ API? > > 4) I saw in kafka.server.KafkaServer has a shutdown method. Is there a > command port that I can send a shutdown command or how to trigger this > shutdown method from an external process such as our process monitor. > > 5) We are thinking if in the future our cluster grow much larger. We > possibly have to add additional kafka broker. What are the parameters we > should look for to determine at that point we should add another broker? > > Thanks in advance for your time and help. > Jamie > >
