Thanks a lot Guozhang! So currently there's no way to delete a consumer group in with the new consumer API? Do you plan to add it in the next versions?
2015-12-31 20:09 GMT+01:00 Guozhang Wang <wangg...@gmail.com>: > Hello Han, > > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to reset > your consuming offsets. Or if you are stopping the consumer between your > test runs you can also commit() with offset 0 before you closing your > consumer at the end of each test. > > 2. In the 0.9 Java consumer both the consumer registry information and the > offsets are stored in Kafka servers instead of on the ZK, you can find a > more detailed description and motivation of this change here: > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design > > So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing the > consumer registry information. > > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see > "deprecations in 0.9.0" here: > > http://kafka.apache.org/documentation.html#upgrade_9_breaking > > Instead you can try to use bin/kafka-consumer-groups.sh > (kafka.admin.ConsumerGroupCommand). > > Guozhang > > > > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju.han.fe...@gmail.com> wrote: > > > Hi Marko, > > > > Yes we're currently using this on our production kafka 0.8. But it does > not > > seem to work with the new consumer API in 0.9. > > To answer my own question about deleting consumer group in new consumer > > API, it seems that it's currently not possible with the new consumer API > > (there's no delete related method in `AdminClient` of the new consumer > > API). > > > > > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <marko.bon...@sematext.com>: > > > > > If you want to monitor offset (ZK or Kafka based), try with QuantFind's > > > Kafka Offset Monitor. > > > If you use Docker, it's easy as: > > > > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181 > > > jpodeszwik/kafka-offset-monitor > > > and then opening browser to dockerhost:8080. > > > > > > If not in the Docker mood, use instructions here: > > > https://github.com/quantifind/KafkaOffsetMonitor > > > > > > Marko Bonaći > > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > > > Solr & Elasticsearch Support > > > Sematext <http://sematext.com/> | Contact > > > <http://sematext.com/about/contact.html> > > > > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju.han.fe...@gmail.com> > wrote: > > > > > > > Thanks guys. The `seek` seems a solution. But it's more cumbersome > than > > > in > > > > 0.8 because I have to plug in some extra code in my consumer > > abstractions > > > > rather than simply deleting a zk node. > > > > And one more question: where does kafka 0.9 stores the consumer-group > > > > information? In fact I also tried to delete the consumer group but > the > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. > And > > > also > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper > > > > localhost:2181 --group group-name` seems broken. > > > > > > > > Thanks! > > > > > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <marko.bon...@sematext.com>: > > > > > > > > > I was refering to Dana Powers's answer in the link I posted (to > use a > > > > > client API). You can find an example here: > > > > > > > > > > > > > > > > > > > > http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > > > > > > > > Marko Bonaći > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log > > Management > > > > > Solr & Elasticsearch Support > > > > > Sematext <http://sematext.com/> | Contact > > > > > <http://sematext.com/about/contact.html> > > > > > > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ssla...@gmail.com> > > > wrote: > > > > > > > > > > > Then I guess @Before test, explicitly commit offset of 0. > > > > > > > > > > > > There doesn't seem to be a tool for committing offset, only for > > > > > > checking/fetching current offset (see > > > > > > http://kafka.apache.org/documentation.html#operations ) > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju.han.fe...@gmail.com> > > > > wrote: > > > > > > > > > > > > > Hi Stevo, > > > > > > > > > > > > > > But by deleting and recreating the topic, do I remove also the > > > > messages > > > > > > > ingested? > > > > > > > My use case is that I ingest prepared messages once and run > > > consumer > > > > > > tests > > > > > > > multiple times, between each test run I reset the consumer > > group's > > > > > offset > > > > > > > so that each run starts from the beginning and consumers all > the > > > > > > messages. > > > > > > > > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ssla...@gmail.com>: > > > > > > > > > > > > > > > Have you considered deleting and recreating topic used in > test? > > > > > > > > Once topic is clean, read/poll once - any committed offset > > should > > > > be > > > > > > > > outside of the range, and consumer should reset offset. > > > > > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU < > > ju.han.fe...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > For local test purpose I need to frequently reset offset > for > > a > > > > > > consumer > > > > > > > > > group. In 0.8 I just delete the consumer group's zk node > > under > > > > > > > > > `/consumers`. But with the redesign of the 0.9, how could I > > > > achieve > > > > > > the > > > > > > > > > same thing? > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *JU Han* > > > > > > > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *JU Han* > > > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *JU Han* > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > -- > > *JU Han* > > > > Software Engineer @ Teads.tv > > > > +33 0619608888 > > > > > > -- > -- Guozhang > -- *JU Han* Software Engineer @ Teads.tv +33 0619608888
