Currently, we have tools like ImportOffset that can update the starting offset of a consumer in ZK. However, one has to stop all consumers first.
We do plan to allow the high level consumer to specify a starting offset in the future when we revisit the consumer design. Some of the details are described in https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design Thanks, Jun On Tue, Apr 2, 2013 at 6:56 AM, James Englert <jengl...@gilt.com> wrote: > I wrote some code > here< > http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E > >that > resets the offset. You just need to be sure to set this as the > consumer(s) is starting up. I'm not certain if it is the "right way" to > do things. > > > http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+hRY70p8NP=spTEm8zaBOEqKcTiVFB=oo...@mail.gmail.com%3E > > > On Tue, Apr 2, 2013 at 12:46 AM, Jun Rao <jun...@gmail.com> wrote: > > > Sean, > > > > A broker can have multiple topics, each with multiple partitions. Each > > partition can be consumed by multiple consumers. > > > > Our high level consumer API doesn't allow you to specify a starting > offset. > > SimpleConsumer does. If you use SimpleConsumer, you are responsible for > > managing the consumption of all partitions. Consumer group is only used > in > > the high level consumer. > > > > Thanks, > > > > Jun > > > > On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson <s...@burstly.com> wrote: > > > > > Hello, > > > > > > Hopefully I'm sending this question to the right place. I'm currently > > > trying to set up a consumer that will allow me to specify the offset, > > > partition, and consumer group ID all at the same time. This obviously > > > causes a dilemma since neither the low-level or high-level consumer > APIs > > > seem to support all three. So I have a couple questions: > > > > > > 1) Am I correct in my understanding that there is a 1-to-1 relationship > > > between brokers and partitions? Are multiple consumers per partition > > > supported? If not, I imagine I could just start up a simple consumer > per > > > partition and that would give me what I want. If so, then do simple > > > consumers actually support consumer group IDs without my knowledge? > > > > > > 2) Is it possible to specify a custom offset for the high-level > consumer > > > API other than from the tail (autooffset.reset = "smallest") or the > head > > > (autooffset.reset = "largest")? I know that the offset for each > consumer, > > > topic, and partition relationship is stored in a zookeeper ephemeral > > node. > > > Would connecting to zookeeper and changing all these offsets to the > > desired > > > values give me what I want? I have tried this with just one consumer > > using > > > the high-level api with one partition and one zookeeper host and it > > seemed > > > to work alright. But I'm wondering if changing these offsets would work > > > correctly and efficiently with multiple partitions and consumers with > the > > > same ID, as well as multiple zookeeper hosts. > > > > > > Thanks, > > > Sean > > > > > > > > > -- > Jim Englert > Gilt Groupe > 2 Park Ave South, 5th Floor > New York, NY 10011 > M: 847-707-2942 > Please accept my invitation to join Gilt: > http://www.giltgroupe.com/invite/jenglert >
