Created this to follow up: https://issues.apache.org/jira/browse/KAFKA-3059
On Mon, Jan 4, 2016 at 9:21 AM, Stevo Slavić <ssla...@gmail.com> wrote: > IMO just command/tool to commit offset for given consumer group and topic > should work (maybe partition can be optional parameter, if partition is not > provided commit same offset for all partitions of given topic). Then one > can commit offset -1 for given consumer group and given topic, effectively > making offset invalid (outside of range). > > On Mon, Jan 4, 2016 at 6:00 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > What you really want is to remove the committed offsets in Kafka servers, > > not the consumer group registry metadata (e.g. such as partition > assignment > > information, etc), which will not be done immediately after the group is > > removed, but only after the offsets expiration has elapsed. > > > > We can add this feature as an admin command in future versions. > > > > Guozhang > > > > > > On Mon, Jan 4, 2016 at 2:12 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > > > Created related ticket > https://issues.apache.org/jira/browse/KAFKA-3057 > > > Maybe request for extra operations support/tools could be a separate > > > ticket. > > > > > > On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju.han.fe...@gmail.com> > wrote: > > > > > > > Thanks a lot Guozhang! > > > > So currently there's no way to delete a consumer group in with the > new > > > > consumer API? Do you plan to add it in the next versions? > > > > > > > > > > > > 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wangg...@gmail.com>: > > > > > > > > > Hello Han, > > > > > > > > > > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer > to > > > > reset > > > > > your consuming offsets. Or if you are stopping the consumer between > > > your > > > > > test runs you can also commit() with offset 0 before you closing > your > > > > > consumer at the end of each test. > > > > > > > > > > 2. In the 0.9 Java consumer both the consumer registry information > > and > > > > the > > > > > offsets are stored in Kafka servers instead of on the ZK, you can > > find > > > a > > > > > more detailed description and motivation of this change here: > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design > > > > > > > > > > So the "AdminUtils.deleteConsumerGroupInZK" will not help in > removing > > > the > > > > > consumer registry information. > > > > > > > > > > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see > > > > > "deprecations in 0.9.0" here: > > > > > > > > > > http://kafka.apache.org/documentation.html#upgrade_9_breaking > > > > > > > > > > Instead you can try to use bin/kafka-consumer-groups.sh > > > > > (kafka.admin.ConsumerGroupCommand). > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju.han.fe...@gmail.com> > > > wrote: > > > > > > > > > > > Hi Marko, > > > > > > > > > > > > Yes we're currently using this on our production kafka 0.8. But > it > > > does > > > > > not > > > > > > seem to work with the new consumer API in 0.9. > > > > > > To answer my own question about deleting consumer group in new > > > consumer > > > > > > API, it seems that it's currently not possible with the new > > consumer > > > > API > > > > > > (there's no delete related method in `AdminClient` of the new > > > consumer > > > > > > API). > > > > > > > > > > > > > > > > > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći < > marko.bon...@sematext.com > > >: > > > > > > > > > > > > > If you want to monitor offset (ZK or Kafka based), try with > > > > QuantFind's > > > > > > > Kafka Offset Monitor. > > > > > > > If you use Docker, it's easy as: > > > > > > > > > > > > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181 > > > > > > > jpodeszwik/kafka-offset-monitor > > > > > > > and then opening browser to dockerhost:8080. > > > > > > > > > > > > > > If not in the Docker mood, use instructions here: > > > > > > > https://github.com/quantifind/KafkaOffsetMonitor > > > > > > > > > > > > > > Marko Bonaći > > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log > > > > Management > > > > > > > Solr & Elasticsearch Support > > > > > > > Sematext <http://sematext.com/> | Contact > > > > > > > <http://sematext.com/about/contact.html> > > > > > > > > > > > > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU < > ju.han.fe...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks guys. The `seek` seems a solution. But it's more > > > cumbersome > > > > > than > > > > > > > in > > > > > > > > 0.8 because I have to plug in some extra code in my consumer > > > > > > abstractions > > > > > > > > rather than simply deleting a zk node. > > > > > > > > And one more question: where does kafka 0.9 stores the > > > > consumer-group > > > > > > > > information? In fact I also tried to delete the consumer > group > > > but > > > > > the > > > > > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in > > > 0.9. > > > > > And > > > > > > > also > > > > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker > > > > --zookeeper > > > > > > > > localhost:2181 --group group-name` seems broken. > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći < > > > marko.bon...@sematext.com > > > > >: > > > > > > > > > > > > > > > > > I was refering to Dana Powers's answer in the link I posted > > (to > > > > > use a > > > > > > > > > client API). You can find an example here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > > > > > > > > > > > > > > > > Marko Bonaći > > > > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log > > > > > > Management > > > > > > > > > Solr & Elasticsearch Support > > > > > > > > > Sematext <http://sematext.com/> | Contact > > > > > > > > > <http://sematext.com/about/contact.html> > > > > > > > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić < > > > ssla...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Then I guess @Before test, explicitly commit offset of 0. > > > > > > > > > > > > > > > > > > > > There doesn't seem to be a tool for committing offset, > only > > > for > > > > > > > > > > checking/fetching current offset (see > > > > > > > > > > http://kafka.apache.org/documentation.html#operations ) > > > > > > > > > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU < > > > > ju.han.fe...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Stevo, > > > > > > > > > > > > > > > > > > > > > > But by deleting and recreating the topic, do I remove > > also > > > > the > > > > > > > > messages > > > > > > > > > > > ingested? > > > > > > > > > > > My use case is that I ingest prepared messages once and > > run > > > > > > > consumer > > > > > > > > > > tests > > > > > > > > > > > multiple times, between each test run I reset the > > consumer > > > > > > group's > > > > > > > > > offset > > > > > > > > > > > so that each run starts from the beginning and > consumers > > > all > > > > > the > > > > > > > > > > messages. > > > > > > > > > > > > > > > > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić < > > ssla...@gmail.com > > > >: > > > > > > > > > > > > > > > > > > > > > > > Have you considered deleting and recreating topic > used > > in > > > > > test? > > > > > > > > > > > > Once topic is clean, read/poll once - any committed > > > offset > > > > > > should > > > > > > > > be > > > > > > > > > > > > outside of the range, and consumer should reset > offset. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU < > > > > > > ju.han.fe...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > For local test purpose I need to frequently reset > > > offset > > > > > for > > > > > > a > > > > > > > > > > consumer > > > > > > > > > > > > > group. In 0.8 I just delete the consumer group's zk > > > node > > > > > > under > > > > > > > > > > > > > `/consumers`. But with the redesign of the 0.9, how > > > > could I > > > > > > > > achieve > > > > > > > > > > the > > > > > > > > > > > > > same thing? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > *JU Han* > > > > > > > > > > > > > > > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > > > > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > *JU Han* > > > > > > > > > > > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *JU Han* > > > > > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *JU Han* > > > > > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > > *JU Han* > > > > > > > > Software Engineer @ Teads.tv > > > > > > > > +33 0619608888 > > > > > > > > > > > > > > > -- > > -- Guozhang > > >
