Thanks for the pointer! Does that mean I don't need to commit the offset with managing partitions and offset manually?
On Thu, May 7, 2020 at 8:02 PM Chris Toomey <ctoo...@gmail.com> wrote: > If you choose to manually assign topic partitions, then you won't be using > the group protocol to dynamically manage partition assignments and thus > don't have a need to poll or heartbeat at any interval. See "Manual > Partition Assignment" in > > https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > . > > On Thu, May 7, 2020 at 3:52 PM Boyuan Zhang <boyu...@apache.org> wrote: > > > Hi team, > > > > I'm building an application which uses Kafka Consumer APIs to read > messages > > from topics. I plan to manually assign TopicPartitions to my consumer and > > seek a certain offset before starting to read. I'll also materialize the > > last read offset and reuse it when creating the consumer later. > > > > Within my usage, I' curious whether I need to commit offset automatically > > or manually. While going through the doc, it seems like committing offset > > is only important to dynamic assignment. > > > > Another question around manual assignment is, is it still true that I > need > > to call poll() continuously to keep the consumer in the group described > as > > below? > > > > > It is also possible that the consumer could encounter a "livelock" > > > situation where it is continuing to send heartbeats, but no progress is > > > being made. To prevent the consumer from holding onto its partitions > > > indefinitely in this case, we provide a liveness detection mechanism > > using > > > the max.poll.interval.ms setting. Basically if you don't call poll at > > > least as frequently as the configured max interval, then the client > will > > > proactively leave the group so that another consumer can take over its > > > partitions. When this happens, you may see an offset commit failure (as > > > indicated by a CommitFailedException > > > < > > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html > > > > thrown > > > from a call to commitSync() > > > < > > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync-- > > >). > > > This is a safety mechanism which guarantees that only active members of > > the > > > group are able to commit offsets. So to stay in the group, you must > > > continue to call poll. > > > > What will happen to poll() with manual assignment if the consumer is > > removed from the group? > > > > Thanks for your help! > > > > Boyuan > > >
