Thank you, Michal. That answers all my questions, many thanks.
Josh On Thu, Oct 5, 2017 at 1:21 PM, Michal Michalski < michal.michal...@zalando.ie> wrote: > Hi Josh, > > 1. I don't know for sure (haven't seen the code that does it), but it's > probably the most "even" split possible for given number of brokers and > partitions. So for 8 partitions and 3 brokers it would be [3, 3, 2]. > 2. See "num.partitions" in broker config. BTW. only producer can create > topic dynamically, not consumer. > 3. See 3. The value has to be non-zero, so it's always specified. > 4. Based on the ProducerRecord (message) key. See: > https://kafka.apache.org/0110/javadoc/index.html?org/apache/ > kafka/clients/producer/KafkaProducer.html > 5. Yes - you need to create multiple consumers with the same group.id. > 6. Yes, there'll be at most one consumer (within a consumer group) handling > given partition at a given time. > 7. Yes, it's a process called "rebalancing" - it reassigns partitions to > consumers when the number of consumers changes. > 8. Your consumer will commit the last processed offset to special Kafka > topic (or Zookeeper, but that's not a default) every so often (periodically > or "on demand", when you tell it to), so for each partition and consumer > group you know what was and wasn't processed yet. The new consumer will > pick up from the place where the dead one left off. > 9. If I understand your question correctly - no, Kafka is pull-based and > not push-based by design. > > Kind regards, > Michał > > On 5 October 2017 at 09:37, Josh Maidana <joshmaid...@gmail.com> wrote: > > > Hello > > > > I am quite new to KAFKA and come from a JMS/messaging background. Reading > > through the documentation, I gather using partitions and consumer groups, > > KAFKA achieves both P2P and pub/sub. I have a few questions on > partitions, > > though, I was wondering someone could kindly please point me in the right > > directions. > > > > 1. In a multi-server scenario, how does KAFKA decide how many partitions > of > > a given topic is assigned to a given node? > > 2. When a topic is created dynamically by a consumer or a producer, how > is > > the number of partitions specified? > > 3. If it is not or can't be specified, how does KAFKA decide the number > of > > partitions to create? > > 4. If a producer doesn't specify a partition, how does KAFKA decide to > > which partition the message is allocated. > > 5. On consumption, do I need to explicitly create multiple consumers to > > attain parallelism? > > 6. If yes, would KAFKA allocate different partition to different > consumers > > who are part of the same consumer group? > > 7. If one of those consumers exit, would KAFKA reallocate the partitions > to > > remaining consumers? > > 8. How are the offsets propagated from an exited to consumer to the new > > consumer to which the partition is reallocated? > > 9. Is there a listener based API for consumption instead os a blocking > > poll? > > > > Kind regards > > Josh > > >
