Does anyone know about this? Altering topic partitions seems to reset consumer offsets.
On Tue, Sep 27, 2016 at 1:28 PM, Juho Autio <juho.au...@rovio.com> wrote: > I increased partitions for one existing topic (2->10), but was surprised > to see that it entirely reset the committed offsets of my consumer group. > > All topics & partitions were reset to the earliest offset available, and > the consumer read everything again. > > Documentation doesn't mention anything like this. Is this how it's > supposed to work, or a bug? > > I would've expected the consumer offsets to not decrease at all, > especially for the topics that I didn't even touch. > > For the altered topic I would've expected that consuming the previously > existing partitions 0 and 1 would've continued from the position where they > were, and naturally starting to read the new added partitions from 0. > > I added partitions according to the "Modifying topics" section of Kafka > 0.10.0 Documentation: > > "To add partitions you can do > > > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic > altered_topic --partitions 10 > " > > Previously this topic had 2 partitions. > > For the consumer I'm using: > kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter() > > And version is: > > <groupId>org.apache.kafka</groupId> > <artifactId>kafka_2.11</artifactId> > <version>0.10.0.1</version> > > Kafka cluster itself is kafka_2.11-0.10.0.1. >
