Hi, On Fri, Dec 6, 2013 at 6:32 PM, Steven Parkes <smpar...@smparkes.net> wrote:
> On Dec 6, 2013, at 2:03 PM, Otis Gospodnetic <otis.gospodne...@gmail.com> > wrote: > > > but I think the > > problem is that each time we grab we could get some of the same messages > we > > already processed > > Doesn't setting the fetchSize to "how far back we need to grab" handle > that? I *think* it doesn't, but I'm wrong every day.... N times, so.... I think this is what would happen: 1) imagine 10 messages in the broker m1 - m10 2) consumer grabs last N (=3): m8, m9, m10 3) while it's doing that and before consumer polls for more messages producer publishes 2 more: m11 and m12 4) consumer now polls again. It asks broker for publisher offset and gets the answer: 12 5) good, says consumer, let me then fetch everything after offset 12-3=9: m10, m11, m12 Problem: consumer got m10 again, but it was already processed in 2). No? Please correct me if I'm wrong anywhere. Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/
