On Thu, Aug 29, 2013 at 1:28 PM, Mark <static.void....@gmail.com> wrote:
> I'm assuming for this auto-detection of partitions is only with the > ZookeeperConsumerConnector and not the ConsumerConnector? > Plain ol' ConsumerConnector does it for us, in our Java-based high-level consumers. > > On Aug 29, 2013, at 1:04 PM, Philip O'Toole <phi...@loggly.com> wrote: > > > On Thu, Aug 29, 2013 at 11:09 AM, Mark <static.void....@gmail.com> > wrote: > > > >> 1) Should a producer be aware of which broker to write to or is this > >> somehow managed by Kafka itself. For example, If I have 2 brokers with a > >> configured partition size of 1 will my messages be written in a > round-robin > >> type of fashion to each broker or is this completed dependent on which > >> broker the consume connects to? > >> > > > > Sounds like a question regarding the Producer, which I am not familiar > > with. We use our own Producers here, written in C++. > > > > > >> > >> 2) Similar question goes for a writing to a topic that has multiple > >> partitions. Are the writes randomly load-balanced between partitions or > is > >> this something the client needs to do? > >> > > > > It depends. Producers can specify the partition, or use a partition of > > '-1', which instructs Kafka to randomly choose a partition. > > > > > >> > >> 3) If I have 1 topic with 1 partition and 2 brokers how does consuming > >> messages work. For example, if my consumer connects to broker 1 how > does it > >> consume messages written to broker 2? > >> > > > > Don't think of partitions on one broker and partitions on another broker. > > Simply think of the aggregate partition count across the cluster. The > > high-level consumer that comes with Kafka will learn where all the > > partitions are for a given topic via Zookeeper, create connections to the > > Kafka brokers as needed, and drain all the partitions. > > > > > >> > >> Thanks > >> > >> > >
