Created related ticket https://issues.apache.org/jira/browse/KAFKA-3057 Maybe request for extra operations support/tools could be a separate ticket.
On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju.han.fe...@gmail.com> wrote: > Thanks a lot Guozhang! > So currently there's no way to delete a consumer group in with the new > consumer API? Do you plan to add it in the next versions? > > > 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wangg...@gmail.com>: > > > Hello Han, > > > > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to > reset > > your consuming offsets. Or if you are stopping the consumer between your > > test runs you can also commit() with offset 0 before you closing your > > consumer at the end of each test. > > > > 2. In the 0.9 Java consumer both the consumer registry information and > the > > offsets are stored in Kafka servers instead of on the ZK, you can find a > > more detailed description and motivation of this change here: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design > > > > So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing the > > consumer registry information. > > > > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see > > "deprecations in 0.9.0" here: > > > > http://kafka.apache.org/documentation.html#upgrade_9_breaking > > > > Instead you can try to use bin/kafka-consumer-groups.sh > > (kafka.admin.ConsumerGroupCommand). > > > > Guozhang > > > > > > > > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju.han.fe...@gmail.com> wrote: > > > > > Hi Marko, > > > > > > Yes we're currently using this on our production kafka 0.8. But it does > > not > > > seem to work with the new consumer API in 0.9. > > > To answer my own question about deleting consumer group in new consumer > > > API, it seems that it's currently not possible with the new consumer > API > > > (there's no delete related method in `AdminClient` of the new consumer > > > API). > > > > > > > > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <marko.bon...@sematext.com>: > > > > > > > If you want to monitor offset (ZK or Kafka based), try with > QuantFind's > > > > Kafka Offset Monitor. > > > > If you use Docker, it's easy as: > > > > > > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181 > > > > jpodeszwik/kafka-offset-monitor > > > > and then opening browser to dockerhost:8080. > > > > > > > > If not in the Docker mood, use instructions here: > > > > https://github.com/quantifind/KafkaOffsetMonitor > > > > > > > > Marko Bonaći > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log > Management > > > > Solr & Elasticsearch Support > > > > Sematext <http://sematext.com/> | Contact > > > > <http://sematext.com/about/contact.html> > > > > > > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju.han.fe...@gmail.com> > > wrote: > > > > > > > > > Thanks guys. The `seek` seems a solution. But it's more cumbersome > > than > > > > in > > > > > 0.8 because I have to plug in some extra code in my consumer > > > abstractions > > > > > rather than simply deleting a zk node. > > > > > And one more question: where does kafka 0.9 stores the > consumer-group > > > > > information? In fact I also tried to delete the consumer group but > > the > > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. > > And > > > > also > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker > --zookeeper > > > > > localhost:2181 --group group-name` seems broken. > > > > > > > > > > Thanks! > > > > > > > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <marko.bon...@sematext.com > >: > > > > > > > > > > > I was refering to Dana Powers's answer in the link I posted (to > > use a > > > > > > client API). You can find an example here: > > > > > > > > > > > > > > > > > > > > > > > > > > > http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > > > > > > > > > > Marko Bonaći > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log > > > Management > > > > > > Solr & Elasticsearch Support > > > > > > Sematext <http://sematext.com/> | Contact > > > > > > <http://sematext.com/about/contact.html> > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ssla...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Then I guess @Before test, explicitly commit offset of 0. > > > > > > > > > > > > > > There doesn't seem to be a tool for committing offset, only for > > > > > > > checking/fetching current offset (see > > > > > > > http://kafka.apache.org/documentation.html#operations ) > > > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU < > ju.han.fe...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi Stevo, > > > > > > > > > > > > > > > > But by deleting and recreating the topic, do I remove also > the > > > > > messages > > > > > > > > ingested? > > > > > > > > My use case is that I ingest prepared messages once and run > > > > consumer > > > > > > > tests > > > > > > > > multiple times, between each test run I reset the consumer > > > group's > > > > > > offset > > > > > > > > so that each run starts from the beginning and consumers all > > the > > > > > > > messages. > > > > > > > > > > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ssla...@gmail.com>: > > > > > > > > > > > > > > > > > Have you considered deleting and recreating topic used in > > test? > > > > > > > > > Once topic is clean, read/poll once - any committed offset > > > should > > > > > be > > > > > > > > > outside of the range, and consumer should reset offset. > > > > > > > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU < > > > ju.han.fe...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > For local test purpose I need to frequently reset offset > > for > > > a > > > > > > > consumer > > > > > > > > > > group. In 0.8 I just delete the consumer group's zk node > > > under > > > > > > > > > > `/consumers`. But with the redesign of the 0.9, how > could I > > > > > achieve > > > > > > > the > > > > > > > > > > same thing? > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *JU Han* > > > > > > > > > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *JU Han* > > > > > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *JU Han* > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > -- > > > *JU Han* > > > > > > Software Engineer @ Teads.tv > > > > > > +33 0619608888 > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > *JU Han* > > Software Engineer @ Teads.tv > > +33 0619608888 >
