Thanks, now I understand largest.
On Wed, Dec 17, 2014 at 5:42 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > > When you add a new consumer group, you can choose where it will start > reading. > > With high-level consumer you can set auto.offset.reset to "largest" > (new messages only) or "smallest" (all messages) , with simple > consumer you can pick specific offsets. > > > On Wed, Dec 17, 2014 at 2:33 PM, Greg Lloyd <gllo...@gmail.com> wrote: > > Thanks for the reply, > > > > So if I wanted to add a new group of consumers 6 months into the lifespan > > of my implementation and I didn't want that new group to process all the > > last six months is there a method to manage this? > > > > > > > > On Tue, Dec 16, 2014 at 9:48 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > >> > >> " If all > >> the consumers stop listening how long will Kafka continue to store > messages > >> for that group?" > >> > >> Kafka retains data for set amount of time, regardless of whether > >> anyone is listening or not. This amount of time is configurable. > >> Because Kafka performance is generally constant with the amount of > >> data it stores, storing terabytes of data for months is not uncommon. > >> > >> Gwen > >> > >> On Tue, Dec 16, 2014 at 1:09 PM, Greg Lloyd <gllo...@gmail.com> wrote: > >> > Hi, > >> > > >> > I am planning to use kafka for a work queue type use case with > multiple > >> > consumers. I also plan to use it with multiple consumer groups. What > is > >> not > >> > clear to me from the documentation is how to define+manage consumer > >> groups > >> > or if that is possible at all. It appears to me that a consumer group > is > >> > created just by connecting with a new group-id. What is not clear to > me > >> is > >> > how to manage this: if I no longer want a consumer group how do I > remove > >> > it? What happens when there are no consumers of a group listening? If > all > >> > the consumers stop listening how long will Kafka continue to store > >> messages > >> > for that group? > >> > > >> > Thanks for any clarification, > >> > > >> > Greg Lloyd > >> >
