If you are using the console consumer to check the offsets topic, remember that you need this line in consumer.properties: exclude.internal.topics=false
On Tue, Sep 22, 2015 at 6:05 AM Joris Peeters <j.peet...@wintoncapital.com> wrote: > Ah, nice! Does not look like it is working, though. For some reason the > __consumer_offsets topic is still empty. I see there's a few debug(..) > logging messages that might get displayed if things go wrong - would you > know how to get those displayed? (Right now I'm just running as 'java -jar > ...' with no logging config). > > -J > > -----Original Message----- > From: tao xiao [mailto:xiaotao...@gmail.com] > Sent: 22 September 2015 10:51 > To: users@kafka.apache.org > Subject: Re: committing offsets > > 0.8.2.1 already supports Kafka offset storage. You can set > offsets.storage=kafka in consumer properties and high level API is able to > pick it up and commit offsets to Kafka > > Here is the code reference where kafka offset logic kicks in > > https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L308 > > On Tue, 22 Sep 2015 at 17:44 Joris Peeters <j.peet...@wintoncapital.com> > wrote: > > > I'm trying to set up a kafka consumer (in Java) that uses the new > > approach of committing offsets (i.e. in the __consumer_offsets topic > > etc, rather than through zookeeper). > > > > Am I correct in believing that the current version (we're using kafka > > 8.2.1) does not expose this through the high level consumer yet? > > > > I am using kafka.consumer.Consumer.createJavaConsumerConnector to set > > up the consumer. I can play with the enabling/disabling of > > auto-commits, or use commitOffsets to commit manually, but that does > > not seem to lead to any entries in __consumer_offsets. Also, by > > following the code trail a bit, it does seem as if offset committing is > still happening through Zookeeper. > > > > (One reason we'd like to use the new system is because we're > > interested in monitoring it through Burrow) > > > > Apologies if I missed something simple. > > Cheers! > > -Joris. > > > > > > > > Winton Capital Management Limited ("Winton") is a limited company > > registered in England and Wales with its registered offices at 16 Old > > Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is > > authorised and regulated by the Financial Conduct Authority in the > > United Kingdom, registered as an investment adviser with the US > > Securities and Exchange Commission, registered with the US Commodity > > Futures Trading Commission and a member of the National Futures > > Association in the United States. > > > > This communication, including any attachments, is confidential and may > > be privileged. This email is for use by the intended recipient only. > > If you receive it in error, please notify the sender and delete it. > > You should not copy or disclose all or any part of this email. > > > > This email does not constitute an offer or solicitation and nothing > > contained in this email constitutes, and should not be construed as, > > investment advice. Prospective investors should request offering > > materials and consult their own advisers with respect to investment > > decisions and inform themselves as to applicable legal requirements, > > exchange control regulations and taxes in the countries of their > > citizenship, residence or domicile. Past performance is not indicative > of future results. > > > > Winton takes reasonable steps to ensure the accuracy and integrity of > > its communications, including emails. However Winton accepts no > > liability for any materials transmitted. Emails are not secure and > > cannot be guaranteed to be error free. > > > > > Winton Capital Management Limited (“Winton”) is a limited company > registered in England and Wales with its registered offices at 16 Old > Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is > authorised and regulated by the Financial Conduct Authority in the United > Kingdom, registered as an investment adviser with the US Securities and > Exchange Commission, registered with the US Commodity Futures Trading > Commission and a member of the National Futures Association in the United > States. > > This communication, including any attachments, is confidential and may be > privileged. This email is for use by the intended recipient only. If you > receive it in error, please notify the sender and delete it. You should not > copy or disclose all or any part of this email. > > This email does not constitute an offer or solicitation and nothing > contained in this email constitutes, and should not be construed as, > investment advice. Prospective investors should request offering materials > and consult their own advisers with respect to investment decisions and > inform themselves as to applicable legal requirements, exchange control > regulations and taxes in the countries of their citizenship, residence or > domicile. Past performance is not indicative of future results. > > Winton takes reasonable steps to ensure the accuracy and integrity of its > communications, including emails. However Winton accepts no liability for > any materials transmitted. Emails are not secure and cannot be guaranteed > to be error free. >
