Hi Mike, Kafka Streams is suitable for distributed environment. You can read this section of the web docs for more details:
http://docs.confluent.io/2.1.0-alpha1/streams/architecture.html#parallelism-model As for the Kafka Streams' customized partition assignor under the Kafka group management protocol, you can find more details here: http://docs.confluent.io/2.1.0-alpha1/streams/developer-guide.html#partition-grouper In the long term, the subscription that coordinator sends to the selected leader includes metadata about the current tasks information, including which instances are hosting these tasks (if it is the first time a rebalance is triggered, then it is empty). The assignor first triggers the user-customizable partition grouper given the list of partitions to get the new list of tasks, then tries to assign tasks to the known "host" (identified by UUID in the code), with the principle that: 1) If the task exists already (i.e. it is not newly created from the partition grouper), then tries to assign it to the current hosting instance if the host is still alive. 2) otherwise if the task has standby replicas on other hosts, then try to assign it to the hosting instance of the standby replicas. 3) otherwise tries to assign to the least-loaded hosting instance. Then the leader groups the task -> host mapping into the partition -> consumer-member mapping (note that a single host, i.e. machine, can have multiple consumers), and sends it back to the coordinator, which will then be propagated to other members. Upon receiving the assignment, these instances will get the tasks from the metadata and the assigned partitions, and start initializing / running the tasks. Guozhang On Sun, Mar 20, 2016 at 7:12 AM, Michael D. Coon <mdco...@yahoo.com.invalid> wrote: > I'm evaluating whether the KafkaStreams API will be something we can use > on my current project. Namely, we want to be able to distribute the > consumers on a Mesos/YARN cluster. It's not entirely clear to me in the > code what is deciding which partitions get assigned at runtime and whether > this is intended for a distributed application or just a multi-threaded > environment. > I get that the consumer coordinator will get reassignments when group > participation changes; however, in looking through the > StreamPartitionAssignor code, it's not clear to me what is happening in the > assign method. It looks like to me like subscriptions are coming in from > the consumer coordinator, presumably whose assignments are derived from the > lead brokers for the topics of interest. Those subscriptions are then > translated into co-partitioned groups of clients. Once that's complete, it > hands off the co-partitioned groups to the StreamThread's partitionGrouper > to do the work of assigning the partitions to each co-partitioned group. > The DefaultPartitionGrouper code, starting on line 57, simply does a 1-up > assigning of partition to group. How will this actually work with > distributed stream consumers if it's always going to be assigning the > partition as a 1-up sequence local to that particular consumer? Shouldn't > it use the assigned partition that is coming back from the > ConsumerCoordinator? I'm struggling to understand the layers but I need to > in order to know whether this implementation is going to work for us. If > the PartitionGroupAssignor's default is just meant for single-node > multithreaded use, that's fine as long as I can inject my own > implementation. But I would still need to understand what is happening at > the StreamPartitionAssignor layer more clearly. Any info, design docs, > in-progress wiki's would be most appreciated if the answer is too in-depth > for an email discussion. Thanks and I really love what you guys are doing > with this! > Mike -- -- Guozhang
