Have you actually examined the Kafka files on disk, to make sure those dupes are really there? Or is this a case of reading the same message more than once?
Philip On Thu, Jul 18, 2013 at 8:55 AM, Sybrandy, Casey <casey.sybra...@six3systems.com> wrote: > Hello, > > We recently started seeing duplicate messages appearing at our consumers. > Thankfully, the database is set up so that we don't store the dupes, but it > is annoying. It's not every message, only about 1% of them. We are running > 0.7.0 for the broker with Zookeeper 3.3.4 from Cloudera and 0.7.0 for the > producer and consumer. We tried upgrading the consumer to 0.7.2 to see if > that worked, but we're still seeing the dupes. Do we have to upgrade the > broker as well to resolve this? Is there something we can check to see > what's going on because we're not seeing anything unusual in the logs. I > suspected that there may be significant rebalancing, but that does not appear > to be the case at all. > > Casey Sybrandy >
