Hi Alexi Typically you would use a key to guarantee that messages with the same key have a global ordering, rather than using manual assignment. Kafka will send all messages with the same key to the same partition. If you need global ordering, spanning all messages from a single producer, you can use a single partition topic. This will limit you to one active consumer per consumer group as the consumer group protocol guarantees that a partition can only be assigned to one consumer, within a group, at one time.
B On Wed, Dec 21, 2016 at 4:36 AM Alexei Levashov < alexei.levas...@arrayent.com> wrote: > Hello, > I have a few newbie questions about usage of Kafka as a messaging system. > Kafka version - 0.10.1.0. > > 1 - Let's assume that I want to ensure time sequence of events i.e. if > message A from producer was published at time t1 to partition P and message > B from the same producer published to partition P at time t2, > I want to consume message A before message B, provided t1<t2. > > Question1, > Do I have any choice except one consumer per partition? > > 2. - If I have one consumer per partition and use > consumer.assign(partitionList) call to assign consumer to a partition do I > still need group membership for this single consumer? > I didn't find clear description what is the protocol of interaction > between GroupCoordinator and PartitionLeader > < > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal > > > will be in case of "manual" partition assignment. > On one hand the API documentation > < > https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html > > > says > that : > "Manual partition assignment does not use group coordination, so > consumer failures will not cause assigned partitions to be rebalanced. > Each consumer acts independently even if it shares a groupId with > another consumer. > To avoid offset commit conflicts, you should usually ensure that the > groupId is unique for each consumer instance." > > On the other hand I am still consuming messages in > consumer.poll(timeout) loop and inside this poll() call consumer should > send heartbeats to coordinator. > > Question 2 . > If consumer doesn't send these heartbeats for [*session.timeout.ms > <http://session.timeout.ms>*] period of time should the partition > ownership be revoked or not? > > If no - does it mean I have to use homegrown heartbeats for consumer > state monitoring? How would the application know that the consumer thread > is dead? > If yes - what callback to notify the application can I use? > ConsumerRebalanceListener is available only for group subscription. > > Thank you. >
