Are you using it from Java or Scala? i.e., are you using the
javaapi.SimpleConsumer or kafka.consumer.SimpleConsumer
In 0.8.2 javaapi we explicitly set version 0 of the
OffsetCommitRequest/OffsetFetchRequest which means it will
commit/fetch to/from ZooKeeper only. If you use the scala API you can
create an OffsetCommitRequest with version set to 1 (which will allow
you to commit to Kafka).
Since we are doing an 0.8.2.1 release we will make the above more
consistent. i.e., you can create OffsetCommitRequests with version 1
even from the javaapi. I will be updating the documentation on this to
make it clearer.
Thanks,
Joel
On Thu, Feb 19, 2015 at 02:28:32PM +0000, Suren wrote:
> Joel,
> Looking at SimpleConsumer in the 0.8.2 code, it is using OffsetCommitRequest
> and sending that over to a broker.
> Is the broker storing that in ZK?
> -Suren
>
>
> On Tuesday, February 17, 2015 12:22 PM, Joel Koshy <jjkosh...@gmail.com>
> wrote:
>
>
> Hi Chris,
>
> In 0.8.2, the simple consumer Java API supports committing/fetching
> offsets that are stored in ZooKeeper. You don't need to issue any
> ConsumerMetadataRequest for this. Unfortunately, the API currently
> does not support fetching offsets that are stored in Kafka.
>
> Thanks,
>
> Joel
>
> On Mon, Feb 16, 2015 at 05:02:08PM -0500, Christopher Piggott wrote:
> > Hi,
> >
> > I am still using 0.8.1.1 because of the CPU use concerns.
> >
> > I'm confused about why the SimpleConsumer has:
> >
> > OffsetCommitResponse commitOffsets(OffsetCommitRequest request)
> >
> > and
> >
> > OffsetFetchResponse fetchOffsets(OffsetFetchRequest request)
> >
> > but no way that I can see to issue a ConsumerMetadataRequest, which is
> > what I think when restarting my consumers so that they can begin
> > working where they last left off (in the event that they were stopped
> > for a while then restarted some time later, and new messages had come
> > in).
> >
> > The fetchOffsets() works on time, usually it looks like you send it
> > Earliest or Latest (beginning or end of what's currently in the
> > stream).
> >
> > I realize the documentation says this:
> >
> >
> > > *Downsides of using SimpleConsumer*The SimpleConsumer does require a
> > > significant amount of work not needed in the Consumer Groups:
> > >
> > > 1. You must keep track of the offsets in your application to know
> > >where you left off consuming.
> > >
> > > But that's not really quite true ... not as long as commitOffsets() has
> > > been provided. It seems the SimpleConsumer provides you with a solution
> > > to only one half of the problem of offset management.
> >
> > Using some zookeeper python scripts I wrote I can see that the
> > commitOffsets() is doing its job and writing to
> >
> >
> > /consumers/myGroupId/offsets/myTopic/0
> >
> >
> > That has this value:
> >
> > ('32757408', ZnodeStat(czxid=2211679, mzxid=14779964, ctime=1423777630972,
> > > mtime=1424122117397, version=12568262, cversion=0, aversion=0,
> > > ephemeralOwner=0, dataLength=8, numChildren=0, pzxid=2211679))
> >
> >
> > Now the question is just how to retrieve that - do I really have to
> > have my client connect to ZK directly? If that's the case, future
> > upgrades would break (e.g. 0.8.2 having its own storage for commit
> > watermarks).
> >
> >
> > What was the intent here, and what's the advice on how to proceed
> > being that 0.8.2 is in an iffy state right now?
> >
> >
> > --Chris
>
>
>
>
