Re: Kafka 0.9 Consumer Group

[Wang, Howard]

Hi Jason,

I used the kafka-consumer-groups.sh to check my consumer group :
~/GitHub/kafka/bin/kafka-consumer-groups.sh  --bootstrap-server <my server
name>   ‹group  test.group --describe   --new-consumer .

I ran this command several times after my app was shut down. I always get
"Consumer group `test.group` does not exist or is rebalancing.² response.



I did set the enable.auto.commit to false. Below is how I set my
KafkaConsumer. 

Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
myAppConfig.getKafkaBroker());
        props.put(ConsumerConfig.GROUP_ID_CONFIG,
myAppConfig.getKafkaConsumerGroup());
        props.put(ConsumerConfig.CLIENT_ID_CONFIG,
myAppConfig.getRandomNodeId());
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");
        props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG,
myAppConfig.getKafkaConsumerSessionTimeout());
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
consumer = new KafkaConsumer<>(props);




Thanks.

Howard

On 1/11/16, 12:55 PM, "Jason Gustafson" <ja...@confluent.io> wrote:

>Sorry, wrong property, I meant enable.auto.commit.
>
>-Jason
>
>On Mon, Jan 11, 2016 at 9:52 AM, Jason Gustafson <ja...@confluent.io>
>wrote:
>
>> Hi Howard,
>>
>> The offsets are persisted in the __consumer_offsets topic indefinitely.
>> Since you're using manual commit, have you ensured that
>>auto.offset.reset
>> is disabled? It might also help if you provide a little more detail on
>>how
>> you're verifying that offsets were lost.
>>
>> -Jason
>>
>> On Mon, Jan 11, 2016 at 7:42 AM, Wang, Howard <howard.w...@washpost.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a question regarding the Kafka 0.9 Consumer Group . I manually
>>> commit offsets using the  Kafka 0.9 Consumer created with a consumer
>>>group.
>>>
>>> However, after my app restarted totally from scratch, the consumer
>>>group
>>> seems to lose all the offsets. Is that true that the consumer offsets
>>>are
>>> transient and will be gone after the consumer group has no member and
>>>gets
>>> deleted?
>>>
>>> Thanks.
>>>
>>> Howard
>>> --
>>>  Howard Wang
>>> Engineering - Big Data and Personalization
>>> Washington Post Media
>>>
>>> 1150 15th St NW, Washington, DC 20071
>>> p. 202-334-9195
>>> Email: howard.w...@washpost.com
>>>
>>
>>