When you add a new consumer group, you can choose where it will start reading.
With high-level consumer you can set auto.offset.reset to "largest" (new messages only) or "smallest" (all messages) , with simple consumer you can pick specific offsets. On Wed, Dec 17, 2014 at 2:33 PM, Greg Lloyd <gllo...@gmail.com> wrote: > Thanks for the reply, > > So if I wanted to add a new group of consumers 6 months into the lifespan > of my implementation and I didn't want that new group to process all the > last six months is there a method to manage this? > > > > On Tue, Dec 16, 2014 at 9:48 PM, Gwen Shapira <gshap...@cloudera.com> wrote: >> >> " If all >> the consumers stop listening how long will Kafka continue to store messages >> for that group?" >> >> Kafka retains data for set amount of time, regardless of whether >> anyone is listening or not. This amount of time is configurable. >> Because Kafka performance is generally constant with the amount of >> data it stores, storing terabytes of data for months is not uncommon. >> >> Gwen >> >> On Tue, Dec 16, 2014 at 1:09 PM, Greg Lloyd <gllo...@gmail.com> wrote: >> > Hi, >> > >> > I am planning to use kafka for a work queue type use case with multiple >> > consumers. I also plan to use it with multiple consumer groups. What is >> not >> > clear to me from the documentation is how to define+manage consumer >> groups >> > or if that is possible at all. It appears to me that a consumer group is >> > created just by connecting with a new group-id. What is not clear to me >> is >> > how to manage this: if I no longer want a consumer group how do I remove >> > it? What happens when there are no consumers of a group listening? If all >> > the consumers stop listening how long will Kafka continue to store >> messages >> > for that group? >> > >> > Thanks for any clarification, >> > >> > Greg Lloyd >>
