At the time of first time it occurred, all replicas was in sync. But after restart of clients and brokers, exception started to occur immediately, and replicas becoming out of sync. As explained in the issue, bug related to configuration and timing of records.
On Fri, Aug 25, 2017 at 10:31 AM, Dan Markhasin <minimi...@gmail.com> wrote: > If you run kafka-topics.sh --describe --topic __consumer_offsets, does it > show that all replicas are in sync? > > On 23 August 2017 at 23:11, Murad Mamedov <m...@muradm.net> wrote: > > > Hi David, > > > > Thanks for reply. However, I don't have problem with number of replicas. > I > > have 3 brokers. And topics configured accordingly, especially > > __consumer_offsets > > > > Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 > > Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type= > > producer > > > > And everything was working find for months, until today. > > > > Why would I want changing replication factor? To what value? > > > > On Wed, Aug 23, 2017 at 11:19 PM, David Frederick < > > david.freder...@gmail.com > > > wrote: > > > > > |> NotEnoughReplicasException: Number of insync replicas for partition > > > __consumer_offsets-17 is [1], below required minimum [2] > > > > > > Please refer to > > > https://stackoverflow.com/questions/37960767/how-to- > > > change-the-replicas-of-kafka-topic. > > > Hope it helps! > > > > > > > > > On Aug 23, 2017 5:17 AM, "Murad Mamedov" <m...@muradm.net> wrote: > > > > > > > Hi, > > > > > > > > Did you manage to find the root cause of this issue? > > > > > > > > Same thing happened here. > > > > > > > > Thanks in advance > > > > > > > > On Tue, Jun 13, 2017 at 7:50 PM, Paul van der Linden < > > p...@sportr.co.uk> > > > > wrote: > > > > > > > > > I managed to solve it by: > > > > > - stopping and deleting all data on kafka & zookeeper > > > > > - stopping all consumers and producers > > > > > - starting kafka & zookeeper, waiting till they are up > > > > > - start all consumers & producers, > > > > > > > > > > Is there a better way to do this, without data loss and halting > > > > everything? > > > > > > > > > > On Tue, Jun 13, 2017 at 4:28 PM, Paul van der Linden < > > > p...@sportr.co.uk> > > > > > wrote: > > > > > > > > > > > A few lines of the logs: > > > > > > > > > > > > [2017-06-13 15:25:37,343] INFO [GroupCoordinator 0]: Stabilized > > group > > > > > > summarizer generation 701 (kafka.coordinator.GroupCoordinator) > > > > > > [2017-06-13 15:25:37,345] INFO [GroupCoordinator 0]: Assignment > > > > received > > > > > > from leader for group summarizer for generation 701 > > > (kafka.coordinator. > > > > > > GroupCoordinator) > > > > > > [2017-06-13 15:25:37,345] ERROR [Replica Manager on Broker 0]: > > Error > > > > > > processing append operation on partition __consumer_offsets-17 > > > > > > (kafka.server.ReplicaManager) > > > > > > org.apache.kafka.common.errors.NotEnoughReplicasException: > Number > > of > > > > > > insync replicas for partition __consumer_offsets-17 is [1], below > > > > > required > > > > > > minimum [2] > > > > > > [2017-06-13 15:25:37,345] INFO [GroupCoordinator 0]: Preparing to > > > > > > restabilize group summarizer with old generation 701 > > > > (kafka.coordinator. > > > > > > GroupCoordinator) > > > > > > > > > > > > This keeps happening, for all consumer offsets and all groups, > etc > > > > > > > > > > > > On Tue, Jun 13, 2017 at 4:21 PM, Paul van der Linden < > > > > p...@sportr.co.uk> > > > > > > wrote: > > > > > > > > > > > >> Hi, > > > > > >> > > > > > >> I'm trying to find out how to at least get my kafka working > again. > > > > > >> Something went wrong and kafka has halted to a throughput of 0 > > > > > messages. It > > > > > >> keeps looping on stablizing consumer groups, and erroring on an > > > append > > > > > >> operation to the offset paritions, plus Not enough replicas. > > > > > >> > > > > > >> The weird things is, that after not being able to work this out > I > > > want > > > > > >> pretty brutal (luckily I can afford to loose more messages): > > > > > >> - delete all kafka and zookeeper instances > > > > > >> - updated kafka > > > > > >> - cleared all disk > > > > > >> > > > > > >> Still kafka is in this unrecoverable error. Does anyone have any > > > idea > > > > > how > > > > > >> to fix this? > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > *Murad M* > > > > *M (tr): +90 (533) 4874329* > > > > *E: m...@muradm.net <m...@muradm.net>* > > > > > > > > > > > > > > > -- > > Regards, > > *Murad M* > > *M (tr): +90 (533) 4874329* > > *E: m...@muradm.net <m...@muradm.net>* > > > -- Regards, *Murad M* *M (tr): +90 (533) 4874329* *E: m...@muradm.net <m...@muradm.net>*
