Ah, that actually makes sense. The consumer-groups.sh script only returns offset data when the group is active. The offsets should still be there, but it does seem unfortunate that there's no way to view them. We have KAFKA-3059 which adds some additional capabilities for managing offsets with this script, so maybe we can extend it to provide this feature as well.
-Jason On Mon, Jan 11, 2016 at 11:33 AM, Wang, Howard <howard.w...@washpost.com> wrote: > Hi Jason, > > I used the kafka-consumer-groups.sh to check my consumer group : > ~/GitHub/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <my server > name> ‹group test.group --describe --new-consumer . > > I ran this command several times after my app was shut down. I always get > "Consumer group `test.group` does not exist or is rebalancing.² response. > > > > I did set the enable.auto.commit to false. Below is how I set my > KafkaConsumer. > > Properties props = new Properties(); > props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, > myAppConfig.getKafkaBroker()); > props.put(ConsumerConfig.GROUP_ID_CONFIG, > myAppConfig.getKafkaConsumerGroup()); > props.put(ConsumerConfig.CLIENT_ID_CONFIG, > myAppConfig.getRandomNodeId()); > props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); > props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none"); > props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, > myAppConfig.getKafkaConsumerSessionTimeout()); > props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, > "org.apache.kafka.common.serialization.StringDeserializer"); > props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, > "org.apache.kafka.common.serialization.StringDeserializer"); > consumer = new KafkaConsumer<>(props); > > > > > Thanks. > > Howard > > On 1/11/16, 12:55 PM, "Jason Gustafson" <ja...@confluent.io> wrote: > > >Sorry, wrong property, I meant enable.auto.commit. > > > >-Jason > > > >On Mon, Jan 11, 2016 at 9:52 AM, Jason Gustafson <ja...@confluent.io> > >wrote: > > > >> Hi Howard, > >> > >> The offsets are persisted in the __consumer_offsets topic indefinitely. > >> Since you're using manual commit, have you ensured that > >>auto.offset.reset > >> is disabled? It might also help if you provide a little more detail on > >>how > >> you're verifying that offsets were lost. > >> > >> -Jason > >> > >> On Mon, Jan 11, 2016 at 7:42 AM, Wang, Howard <howard.w...@washpost.com > > > >> wrote: > >> > >>> Hi, > >>> > >>> I have a question regarding the Kafka 0.9 Consumer Group . I manually > >>> commit offsets using the Kafka 0.9 Consumer created with a consumer > >>>group. > >>> > >>> However, after my app restarted totally from scratch, the consumer > >>>group > >>> seems to lose all the offsets. Is that true that the consumer offsets > >>>are > >>> transient and will be gone after the consumer group has no member and > >>>gets > >>> deleted? > >>> > >>> Thanks. > >>> > >>> Howard > >>> -- > >>> Howard Wang > >>> Engineering - Big Data and Personalization > >>> Washington Post Media > >>> > >>> 1150 15th St NW, Washington, DC 20071 > >>> p. 202-334-9195 > >>> Email: howard.w...@washpost.com > >>> > >> > >> > >
