Right -- offset storage is an optional feature with Kafka, you can always choose to not use it and instead manage offsets yourself.
On Thu, May 7, 2020 at 8:07 PM Boyuan Zhang <boyu...@apache.org> wrote: > Thanks for the pointer! Does that mean I don't need to commit the offset > with managing partitions and offset manually? > > On Thu, May 7, 2020 at 8:02 PM Chris Toomey <ctoo...@gmail.com> wrote: > > > If you choose to manually assign topic partitions, then you won't be > using > > the group protocol to dynamically manage partition assignments and thus > > don't have a need to poll or heartbeat at any interval. See "Manual > > Partition Assignment" in > > > > > https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > . > > > > On Thu, May 7, 2020 at 3:52 PM Boyuan Zhang <boyu...@apache.org> wrote: > > > > > Hi team, > > > > > > I'm building an application which uses Kafka Consumer APIs to read > > messages > > > from topics. I plan to manually assign TopicPartitions to my consumer > and > > > seek a certain offset before starting to read. I'll also materialize > the > > > last read offset and reuse it when creating the consumer later. > > > > > > Within my usage, I' curious whether I need to commit offset > automatically > > > or manually. While going through the doc, it seems like committing > offset > > > is only important to dynamic assignment. > > > > > > Another question around manual assignment is, is it still true that I > > need > > > to call poll() continuously to keep the consumer in the group described > > as > > > below? > > > > > > > It is also possible that the consumer could encounter a "livelock" > > > > situation where it is continuing to send heartbeats, but no progress > is > > > > being made. To prevent the consumer from holding onto its partitions > > > > indefinitely in this case, we provide a liveness detection mechanism > > > using > > > > the max.poll.interval.ms setting. Basically if you don't call poll > at > > > > least as frequently as the configured max interval, then the client > > will > > > > proactively leave the group so that another consumer can take over > its > > > > partitions. When this happens, you may see an offset commit failure > (as > > > > indicated by a CommitFailedException > > > > < > > > > > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html > > > > > > thrown > > > > from a call to commitSync() > > > > < > > > > > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync-- > > > >). > > > > This is a safety mechanism which guarantees that only active members > of > > > the > > > > group are able to commit offsets. So to stay in the group, you must > > > > continue to call poll. > > > > > > What will happen to poll() with manual assignment if the consumer is > > > removed from the group? > > > > > > Thanks for your help! > > > > > > Boyuan > > > > > >
