Danny created KAFKA-7348:
----------------------------
Summary: topic partition suddenly rollback
Key: KAFKA-7348
URL: https://issues.apache.org/jira/browse/KAFKA-7348
Project: Kafka
Issue Type: Bug
Affects Versions: 0.9.0.1
Environment: 3 servers
Reporter: Danny
I continue polling topic（HIK_SMART_METADATA_TOPIC），suddenly,this partition goes
rollback low number,as below:
Line 6465: 2018-08-24 15:07:10.359 [vsp-batch-hbase-connector-POLL-0-0] INFO
c.h.b.h.d.connector.v2.dag.impl.KafkaPoller - rbb current topic is
HIK_SMART_METADATA_TOPIC, partition is 11, offset is
{color:#FF0000}39676874{color}
Line 6563: 2018-08-24 15:07:53.151 [vsp-batch-hbase-connector-POLL-0-0] INFO
c.h.b.h.d.connector.v2.dag.impl.KafkaPoller - rbb current topic is
HIK_SMART_METADATA_TOPIC, partition is 11, offset is
{color:#FF0000}35745772{color}
{color:#333333}In this period, leader nodes(all partitions)
swit{color}{color:#333333}ch continuously{color}
{color:#333333}server 1:{color}
{color:#333333}Line 24272: [2018-08-24 15:00:29,555] INFO
[ReplicaFetcherManager on broker 0] Added fetcher for partitions
List([[__consumer_offsets,16], initOffset 0 to broker
BrokerEndPoint(1,17.18.252.211,9092)] , [[__consumer_offsets,28], initOffset 0
to broker BrokerEndPoint(1,17.18.252.211,9092)] ,
[[HUMANBODYCOMPARE_PROGRESS,0], initOffset 1664 to broker
BrokerEndPoint(1,17.18.252.211,9092)] , [[HIK_SMART_METADATA_TOPIC,11],
initOffset 39673107 to broker BrokerEndPoint(1,17.18.252.211,9092)] ,
[[HIK_SMART_METADATA,3], initOffset 0 to broker
BrokerEndPoint(1,17.18.252.211,9092)] , [[__consumer_offsets,4], initOffset 0
to broker BrokerEndPoint(1,17.18.252.211,9092)] ,
[[HIK_SMART_METADATA_NSA_TOPIC,5], initOffset 0 to broker
BrokerEndPoint(1,17.18.252.211,9092)] , [[__consumer_offsets,40], initOffset
70244070 to broker BrokerEndPoint(1,17.18.252.211,9092)] ,
[[HUMANBODYCOMPARE_PROGRESS,6], initOffset 1667 to broker
BrokerEndPoint(1,17.18.252.211,9092)] , [[HIK_SMART_METADATA_JSON_TOPIC,4],
initOffset 0 to broker...
Line 51531: [2018-08-24 15:04:34,568] INFO [ReplicaFetcherManager on broker 0]
Removed fetcher for partitions
{color:#f79232}[HIK_SMART_METADATA_TOPIC,11{color}]
(kafka.server.ReplicaFetcherManager)
Line 51533: [2018-08-24 15:04:34,634] INFO [ReplicaFetcherManager on broker 0]
Added fetcher for partitions List([[HIK_SMART_METADATA_TOPIC,11], initOffset
39675430 to broker BrokerEndPoint(2,XXXX] )
(kafka.server.ReplicaFetcherManager){color}
{color:#333333}server2:{color}
{color:#333333}Line 4954244: [2018-08-24 15:06:13,299] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954245: [2018-08-24 15:06:13,300] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954322: [2018-08-24 15:06:23,303] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954322: [2018-08-24 15:06:23,303] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954323: [2018-08-24 15:06:23,306] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954399: [2018-08-24 15:06:33,317] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954399: [2018-08-24 15:06:33,317] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954400: [2018-08-24 15:06:33,319] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954474: [2018-08-24 15:06:43,303] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954474: [2018-08-24 15:06:43,303] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954475: [2018-08-24 15:06:43,305] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954552: [2018-08-24 15:06:53,340] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954552: [2018-08-24 15:06:53,340] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954553: [2018-08-24 15:06:53,343] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954627: [2018-08-24 15:07:03,317] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954627: [2018-08-24 15:07:03,317] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954628: [2018-08-24 15:07:03,319] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954694: [2018-08-24 15:07:13,315] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954694: [2018-08-24 15:07:13,315] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954695: [2018-08-24 15:07:13,318] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954777: [2018-08-24 15:07:24,133] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954777: [2018-08-24 15:07:24,133] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954778: [2018-08-24 15:07:24,135] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954869: [2018-08-24 15:07:35,775] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954869: [2018-08-24 15:07:35,775] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954870: [2018-08-24 15:07:35,778] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954942: [2018-08-24 15:07:43,308] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954942: [2018-08-24 15:07:43,308] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Shrinking ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 1,0,2 to 1 (kafka.cluster.Partition)
Line 4954943: [2018-08-24 15:07:43,310] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 1: Cached zkVersion [3605] not equal to
that in zookeeper, skip updating ISR (kafka.cluster.Partition)
Line 4954997: [2018-08-24 15:07:52,203] INFO [ReplicaFetcherManager on broker
1] Removed fetcher for partitions [HIK_SMART_METADATA_TOPIC,11]
(kafka.server.ReplicaFetcherManager)
Line 4954999: [2018-08-24 15:07:52,469] INFO [ReplicaFetcherManager on broker
1] Added fetcher for partitions List([[HIK_SMART_METADATA_TOPIC,11], initOffset
39675430 to broker BrokerEndPoint(2,XXXX)] )
(kafka.server.ReplicaFetcherManager){color}
{color:#333333}server 3:{color}
{color:#333333}Line 1191931: kafka.common.NotEnoughReplicasException: Number of
insync replicas for partition [HIK_SMART_METADATA_TOPIC,11] is [1], below
required minimum [2]
Line 1192254: [2018-08-24 15:04:42,892] ERROR [Replica Manager on Broker 2]:
Error processing append operation on partition [HIK_SMART_METADATA_TOPIC,11]
(kafka.server.ReplicaManager)
Line 1192255: kafka.common.NotEnoughReplicasException: Number of insync
replicas for partition [HIK_SMART_METADATA_TOPIC,11] is [1], below required
minimum [2]
Line 1192518: [2018-08-24 15:04:42,904] ERROR [Replica Manager on Broker 2]:
Error processing append operation on partition [HIK_SMART_METADATA_TOPIC,11]
(kafka.server.ReplicaManager)
Line 1192519: kafka.common.NotEnoughReplicasException: Number of insync
replicas for partition [HIK_SMART_METADATA_TOPIC,11] is [1], below required
minimum [2]
Line 1192878: [2018-08-24 15:04:42,917] ERROR [Replica Manager on Broker 2]:
Error processing append operation on partition [HIK_SMART_METADATA_TOPIC,11]
(kafka.server.ReplicaManager)
Line 1192879: kafka.common.NotEnoughReplicasException: Number of insync
replicas for partition [HIK_SMART_METADATA_TOPIC,11] is [1], below required
minimum [2]
Line 1192951: [2018-08-24 15:04:42,920] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 2: Expanding ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 2 to 2,0 (kafka.cluster.Partition)
Line 1192951: [2018-08-24 15:04:42,920] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 2: Expanding ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 2 to 2,0 (kafka.cluster.Partition)
Line 1193050: [2018-08-24 15:07:52,893] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 2: Expanding ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 2,0 to 2,0,1 (kafka.cluster.Partition)
Line 1193050: [2018-08-24 15:07:52,893] INFO Partition
[HIK_SMART_METADATA_TOPIC,11] on broker 2: Expanding ISR for partition
[HIK_SMART_METADATA_TOPIC,11] from 2,0 to 2,0,1 (kafka.cluster.Partition){color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
