Krish, Does broker 0 and 3 have the similar warn log entries as broker 2 for stale controller epochs?
Guozhang On Thu, Jul 9, 2015 at 2:07 PM, Krishna Kumar <kku...@nanigans.com> wrote: > So we tried taking that node down. But that didn¹t fix the issue, so we > restarted the other nodes. > > This seems to have lead to 2 of other replicas dropping out of the ISIR > for *all* topics. > > Topic: topic2 Partition: 0 Leader: 1 Replicas: 1,0,2 Isr: 1 > Topic: topic2 Partition: 1 Leader: 1 Replicas: 2,1,0 > Isr: 1 > Topic: topic2 Partition: 2 Leader: 1 Replicas: 0,2,1 > Isr: 1 > Topic: topic2 Partition: 3 Leader: 1 Replicas: 1,2,0 > Isr: 1 > > > I am seeing this message => Broker 2 ignoring LeaderAndIsr request from > controller 1 with correlation id 8685 since its controller epoch 21 is > old. Latest known controller epoch is 89 (state.change.logger) > > > > On 7/9/15, 4:02 PM, "Krishna Kumar" <kku...@nanigans.com> wrote: > > >Thanks Guozhang > > > >We did do the partition-assignment, but against another topic, and that > >went well. > > > >But this happened for this topic without doing anything. > > > >Regards > >Krish > > > >On 7/9/15, 3:56 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: > > > >>Krishna, > >> > >>Did you run any admin tools after adding the node (I assume it is node > >>3), > >>like partition-assignment? It is shown as the only one in ISR list but > >>not > >>in the replica list, which seems that the partition migration process was > >>not completed. > >> > >>You can verify if this is the case by checking your controller log and > >>see > >>if there are any exception / error entries. > >> > >>Guozhang > >> > >>On Thu, Jul 9, 2015 at 12:04 PM, Krishna Kumar <kku...@nanigans.com> > >>wrote: > >> > >>> Hi > >>> > >>> We added a Kafka node and it suddenly became the leader and the sole > >>> replica for some partitions, but it is not in the ISR > >>> > >>> Any idea how we might be able to fix this? We are on Kafka 0.8.2 > >>> > >>> Topic: topic1 Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: > >>>2,0,1 > >>> Topic: topic1 Partition: 1 Leader: 3 Replicas: 0,2,1 > >>> Isr: 3 > >>> Topic: topic1 Partition: 2 Leader: 3 Replicas: 1,0,2 > >>> Isr: 3 > >>> Topic: topic1 Partition: 3 Leader: 2 Replicas: 2,0,1 > >>> Isr: 2,0,1 > >>> Topic: topic1 Partition: 4 Leader: 3 Replicas: 0,1,2 > >>> Isr: 3 > >>> Topic: topic1 Partition: 5 Leader: 1 Replicas: 1,2,0 > >>> Isr: 1,2,0 > >>> Topic: topic1 Partition: 6 Leader: 3 Replicas: 2,1,0 > >>> Isr: 3 > >>> Topic: topic1 Partition: 7 Leader: 0 Replicas: 0,2,1 > >>> Isr: 0,1,2 > >>> > >>> > >>> > >>> > >>> > > >>> > >>> > >> > >> > >>-- > >>-- Guozhang > > > > -- -- Guozhang
