Eric Abbott created KAFKA-4003:
----------------------------------
Summary: Followers are dropped for being too far behind, yet are
less than 30 messages / 10k bytes behind.
Key: KAFKA-4003
URL: https://issues.apache.org/jira/browse/KAFKA-4003
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.9.0.1
Reporter: Eric Abbott
# Settings are at defaults other than the following
replica.fetch.max.bytes=52428800
num.replica.fetchers=8
num.network.threads=128
Followers are dropped for being too far behind, yet are less than 30 messages /
10k bytes behind. The issue becomes more apparent during periods of high load,
though is not exclusive to high load periods.
We enabled the log4j plugin and turned on DEBUG (for 60s) for
kafka.cluster.Partition
kafka.server.ReplicaManager
The attached DEBUG log data is excerpted for topic,partition
'contacts-pub-sub,7' and shows that reads are happening. However the reads
never read up to the log end offset of the Leader Replica, and thus the
Replica.lastCaughtUpTimeMsUnderlying field is not updated for extended periods
of time, which gets the follower dropped. Not too long after being dropped, the
follower will get picked up again.
[2016-07-28 13:31:36,608] DEBUG [Replica Manager on Broker 157]: Recording
follower broker 158 log read results:
Map([email-raw-sparkpost-events-sox-signed-unknown,9] -> Fetch Data:
[FetchDataInfo(1434 [1286 : 104914],ByteBufferMessageSet())], HW: [1434],
readSize: [50000000], isReadFromLogEnd: [true], error: [None],
[email-events-eventtracking,15] -> Fetch Data: [FetchDataInfo(109074743
[108253610 : 830497723],ByteBufferMessageSet())], HW: [109074743], readSize:
[50000000], isReadFromLogEnd: [true], error: [None], [oauth-authorization,26]
-> Fetch Data: [FetchDataInfo(5468 [5400 : 5469],ByteBufferMessageSet())], HW:
[5468], readSize: [50000000], isReadFromLogEnd: [true], error: [None],
[eventtracking-event-callbacks-c,14] -> Fetch Data: [FetchDataInfo(392319869
[391972118 : 470595480],ByteBufferMessageSet())], HW: [392319869], readSize:
[50000000], isReadFromLogEnd: [true], error: [None],
[emailvalidation-strikeiron-responses,8] -> Fetch Data: [FetchDataInfo(3547001
[3527594 : 19771984],ByteBufferMessageSet())], HW: [3547001], readSize:
[50000000], isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] ->
Fetch Data: [FetchDataInfo(316637965 [316090773 :
831265168],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 286015273, key = java.nio.HeapByteBuffer[pos=0 lim=67 cap=2331], payload
= java.nio.HeapByteBuffer[pos=0 lim=2260 cap=2260]),316637965),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1018327823, key =
java.nio.HeapByteBuffer[pos=0 lim=91 cap=1546], payload =
java.nio.HeapByteBuffer[pos=0 lim=1451 cap=1451]),316637966),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2467233135, key =
java.nio.HeapByteBuffer[pos=0 lim=65 cap=1429], payload =
java.nio.HeapByteBuffer[pos=0 lim=1360 cap=1360]),316637967)))], HW:
[316637962], readSize: [50000000], isReadFromLogEnd: [false], error: [None],
[contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905113 [2834198288 :
966349167],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 1940228036, key = java.nio.HeapByteBuffer[pos=0 lim=12 cap=410], payload
= java.nio.HeapByteBuffer[pos=0 lim=394 cap=394]),2835905113),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1613735028, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=576], payload =
java.nio.HeapByteBuffer[pos=0 lim=558 cap=558]),2835905114),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1573356394, key =
java.nio.HeapByteBuffer[pos=0 lim=16 cap=844], payload =
java.nio.HeapByteBuffer[pos=0 lim=824 cap=824]),2835905115),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3816197841, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=1326], payload =
java.nio.HeapByteBuffer[pos=0 lim=1308 cap=1308]),2835905116),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2906657432, key =
java.nio.HeapByteBuffer[pos=0 lim=15 cap=243], payload =
java.nio.HeapByteBuffer[pos=0 lim=224 cap=224]),2835905117),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1616776650, key =
java.nio.HeapByteBuffer[pos=0 lim=16 cap=845], payload =
java.nio.HeapByteBuffer[pos=0 lim=825 cap=825]),2835905118),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2182693743, key =
java.nio.HeapByteBuffer[pos=0 lim=15 cap=778], payload =
java.nio.HeapByteBuffer[pos=0 lim=759 cap=759]),2835905119),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3254937137, key =
java.nio.HeapByteBuffer[pos=0 lim=15 cap=1375], payload =
java.nio.HeapByteBuffer[pos=0 lim=1356 cap=1356]),2835905120),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1434287683, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=412], payload =
java.nio.HeapByteBuffer[pos=0 lim=394 cap=394]),2835905121)))], HW:
[2835905103], readSize: [50000000], isReadFromLogEnd: [false], error: [None],
[email-events-sox-verified,11] -> Fetch Data: [FetchDataInfo(1149863327
[1149653464 : 125554894],FileMessageSet(MessageAndOffset(Message(magic = 0,
attributes = 2, crc = 758891241, key = null, payload =
java.nio.HeapByteBuffer[pos=0 lim=1249 cap=1249]),1149863327),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2037653886, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1218 cap=1218]),1149863328),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1564011888, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1219 cap=1219]),1149863329),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2958390241, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1271 cap=1271]),1149863330),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 704061401, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1312 cap=1312]),1149863331),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 362204766, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1251 cap=1251]),1149863332),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2874177783, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1212 cap=1212]),1149863333),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1728470591, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1204 cap=1204]),1149863334),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 4127052822, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=16617
cap=16617]),1149863377)))], HW: [1149863270], readSize: [50000000],
isReadFromLogEnd: [false], error: [None], [email-events-sox-verified,3] ->
Fetch Data: [FetchDataInfo(1149891315 [1149636809 :
160829183],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2,
crc = 50411905, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1183
cap=1183]),1149891315), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 3321837456, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1225
cap=1225]),1149891316), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 4128609644, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1219
cap=1219]),1149891317), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 3262264350, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1272
cap=1272]),1149891318), MessageAndOffset(Message(magic = 0, attributes = 0, crc
= 4172597564, key = java.nio.HeapByteBuffer[pos=0 lim=36 cap=685], payload =
java.nio.HeapByteBuffer[pos=0 lim=645 cap=645]),1149891319),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 672906920, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1214 cap=1214]),1149891320),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 4044991181, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=15423
cap=15423]),1149891360), MessageAndOffset(Message(magic = 0, attributes = 2,
crc = 3182323659, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1232
cap=1232]),1149891361)))], HW: [1149891260], readSize: [50000000],
isReadFromLogEnd: [false], error: [None]) (kafka.server.ReplicaManager)
[2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226]
is larger than new hw 2835905103 [2834198288 : 966344226] for partition
[contacts-pub-sub,7]. All leo's are 2835905123 [2834198288 :
966356596],2835905116 [2834198288 : 966351063],2835905103 [2834198288 :
966344226] (kafka.cluster.Partition)
[2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226]
is larger than new hw 2835905103 [2834198288 : 966344226] for partition
[contacts-pub-sub,7]. All leo's are 2835905123 [2834198288 :
966356596],2835905116 [2834198288 : 966351063],2835905103 [2834198288 :
966344226] (kafka.cluster.Partition)
[2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226]
is larger than new hw 2835905103 [2834198288 : 966344226] for partition
[contacts-pub-sub,7]. All leo's are 2835905124 [2834198288 :
966357019],2835905116 [2834198288 : 966351063],2835905103 [2834198288 :
966344226] (kafka.cluster.Partition)
[2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Recorded replica 160 log end offset (LEO) position 2835905116 for partition
[contacts-pub-sub,7]. (kafka.cluster.Partition)
[2016-07-28 13:31:36,618] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226]
is larger than new hw 2835905103 [2834198288 : 966344226] for partition
[contacts-pub-sub,7]. All leo's are 2835905125 [2834198288 :
966358120],2835905116 [2834198288 : 966351063],2835905103 [2834198288 :
966344226] (kafka.cluster.Partition)
[2016-07-28 13:31:36,618] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226]
is larger than new hw 2835905103 [2834198288 : 966344226] for partition
[contacts-pub-sub,7]. All leo's are 2835905126 [2834198288 :
966358532],2835905116 [2834198288 : 966351063],2835905103 [2834198288 :
966344226] (kafka.cluster.Partition)
[2016-07-28 13:31:36,623] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226]
is larger than new hw 2835905103 [2834198288 : 966344226] for partition
[contacts-pub-sub,7]. All leo's are 2835905127 [2834198288 :
966358952],2835905116 [2834198288 : 966351063],2835905103 [2834198288 :
966344226] (kafka.cluster.Partition)
[2016-07-28 13:31:36,626] DEBUG Partition [contacts-pub-sub,7] on broker 157:
High watermark for partition [contacts-pub-sub,7] updated to 2835905113
[2834198288 : 966349167] (kafka.cluster.Partition)
[2016-07-28 13:31:36,642] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905113 [2834198288 : 966349167]
is larger than new hw 2835905113 [2834198288 : 966349167] for partition
[contacts-pub-sub,7]. All leo's are 2835905129 [2834198288 :
966359797],2835905116 [2834198288 : 966351063],2835905113 [2834198288 :
966349167] (kafka.cluster.Partition)
[2016-07-28 13:31:36,642] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905113 [2834198288 : 966349167]
is larger than new hw 2835905113 [2834198288 : 966349167] for partition
[contacts-pub-sub,7]. All leo's are 2835905129 [2834198288 :
966359797],2835905116 [2834198288 : 966351063],2835905113 [2834198288 :
966349167] (kafka.cluster.Partition)
[2016-07-28 13:31:36,645] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Recorded replica 158 log end offset (LEO) position 2835905113 for partition
[contacts-pub-sub,7]. (kafka.cluster.Partition)
[2016-07-28 13:31:36,647] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905113 [2834198288 : 966349167]
is larger than new hw 2835905113 [2834198288 : 966349167] for partition
[contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 :
966360310],2835905116 [2834198288 : 966351063],2835905113 [2834198288 :
966349167] (kafka.cluster.Partition)
[2016-07-28 13:31:36,656] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Lagging replicas for partition [contacts-pub-sub,7] are 160,158
(kafka.cluster.Partition)
[2016-07-28 13:31:36,657] INFO Partition [contacts-pub-sub,7] on broker 157:
Shrinking ISR for partition [contacts-pub-sub,7] from 157,160,158 to 157
(kafka.cluster.Partition)
[2016-07-28 13:31:36,659] DEBUG Partition [contacts-pub-sub,7] on broker 157:
High watermark for partition [contacts-pub-sub,7] updated to 2835905130
[2834198288 : 966360310] (kafka.cluster.Partition)
[2016-07-28 13:31:36,662] DEBUG [Replica Manager on Broker 157]: Recording
follower broker 160 log read results: Map([biden-rss-events,10] -> Fetch Data:
[FetchDataInfo(2772927 [2692556 : 27087325],ByteBufferMessageSet())], HW:
[2772927], readSize: [50000000], isReadFromLogEnd: [true], error: [None],
[cps-es-indexer-a,28] -> Fetch Data: [FetchDataInfo(519301 [496185 :
68076836],ByteBufferMessageSet())], HW: [519301], readSize: [50000000],
isReadFromLogEnd: [true], error: [None],
[billing-amendment-applied-notifications,7] -> Fetch Data: [FetchDataInfo(2557
[2533 : 2856],ByteBufferMessageSet())], HW: [2557], readSize: [50000000],
isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] -> Fetch Data:
[FetchDataInfo(316637968 [316090773 :
831270540],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 1788356129, key = java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload
= java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637968),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1996475757, key =
java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload =
java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637969)))], HW: [316637965],
readSize: [50000000], isReadFromLogEnd: [false], error: [None],
[contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905122 [2834198288 :
966356174],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 3917496555, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload
= java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905122),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3390031010, key =
java.nio.HeapByteBuffer[pos=0 lim=16 cap=401], payload =
java.nio.HeapByteBuffer[pos=0 lim=381 cap=381]),2835905123),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2641803275, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=1079], payload =
java.nio.HeapByteBuffer[pos=0 lim=1061 cap=1061]),2835905124),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 217385418, key =
java.nio.HeapByteBuffer[pos=0 lim=16 cap=390], payload =
java.nio.HeapByteBuffer[pos=0 lim=370 cap=370]),2835905125),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3321979137, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=398], payload =
java.nio.HeapByteBuffer[pos=0 lim=380 cap=380]),2835905126),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3220348147, key =
java.nio.HeapByteBuffer[pos=0 lim=15 cap=401], payload =
java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905127),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2893466760, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload =
java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905128),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2397577902, key =
java.nio.HeapByteBuffer[pos=0 lim=12 cap=491], payload =
java.nio.HeapByteBuffer[pos=0 lim=475 cap=475]),2835905129)))], HW:
[2835905130], readSize: [50000000], isReadFromLogEnd: [false], error: [None],
[email-events-sox-verified,3] -> Fetch Data: [FetchDataInfo(1149891362
[1149636809 : 160852840],FileMessageSet(MessageAndOffset(Message(magic = 0,
attributes = 2, crc = 712083310, key = null, payload =
java.nio.HeapByteBuffer[pos=0 lim=1244 cap=1244]),1149891362)))], HW:
[1149891315], readSize: [50000000], isReadFromLogEnd: [false], error: [None])
(kafka.server.ReplicaManager)
[2016-07-28 13:31:36,671] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310]
is larger than new hw 2835905130 [2834198288 : 966360310] for partition
[contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 : 966360310]
(kafka.cluster.Partition)
[2016-07-28 13:31:36,671] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Recorded replica 160 log end offset (LEO) position 2835905122 for partition
[contacts-pub-sub,7]. (kafka.cluster.Partition)
[2016-07-28 13:31:36,691] ERROR [Replica Manager on Broker 157]: Error
processing append operation on partition [contacts-pub-sub,7]
(kafka.server.ReplicaManager)
kafka.common.NotEnoughReplicasException: Number of insync replicas for
partition [contacts-pub-sub,7] is [1], below required minimum [2]
[2016-07-28 13:31:36,698] ERROR [Replica Manager on Broker 157]: Error
processing append operation on partition [contacts-pub-sub,7]
(kafka.server.ReplicaManager)
kafka.common.NotEnoughReplicasException: Number of insync replicas for
partition [contacts-pub-sub,7] is [1], below required minimum [2]
[2016-07-28 13:31:36,701] ERROR [Replica Manager on Broker 157]: Error
processing append operation on partition [contacts-pub-sub,7]
(kafka.server.ReplicaManager)
kafka.common.NotEnoughReplicasException: Number of insync replicas for
partition [contacts-pub-sub,7] is [1], below required minimum [2]
[2016-07-28 13:31:36,710] ERROR [Replica Manager on Broker 157]: Error
processing append operation on partition [contacts-pub-sub,7]
(kafka.server.ReplicaManager)
kafka.common.NotEnoughReplicasException: Number of insync replicas for
partition [contacts-pub-sub,7] is [1], below required minimum [2]
[2016-07-28 13:31:36,709] DEBUG [Replica Manager on Broker 157]: Recording
follower broker 158 log read results:
Map([email-raw-sparkpost-events-sox-signed-unknown,9] -> Fetch Data:
[FetchDataInfo(1434 [1286 : 104914],ByteBufferMessageSet())], HW: [1434],
readSize: [50000000], isReadFromLogEnd: [true], error: [None],
[email-events-eventtracking,15] -> Fetch Data: [FetchDataInfo(109074743
[108253610 : 830497723],ByteBufferMessageSet())], HW: [109074743], readSize:
[50000000], isReadFromLogEnd: [true], error: [None], [oauth-authorization,26]
-> Fetch Data: [FetchDataInfo(5468 [5400 : 5469],ByteBufferMessageSet())], HW:
[5468], readSize: [50000000], isReadFromLogEnd: [true], error: [None],
[eventtracking-event-callbacks-c,14] -> Fetch Data: [FetchDataInfo(392319869
[391972118 : 470595480],ByteBufferMessageSet())], HW: [392319869], readSize:
[50000000], isReadFromLogEnd: [true], error: [None],
[emailvalidation-strikeiron-responses,8] -> Fetch Data: [FetchDataInfo(3547001
[3527594 : 19771984],ByteBufferMessageSet())], HW: [3547001], readSize:
[50000000], isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] ->
Fetch Data: [FetchDataInfo(316637968 [316090773 :
831270540],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 1788356129, key = java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload
= java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637968),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1996475757, key =
java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload =
java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637969)))], HW: [316637965],
readSize: [50000000], isReadFromLogEnd: [false], error: [None],
[contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905122 [2834198288 :
966356174],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 3917496555, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload
= java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905122),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3390031010, key =
java.nio.HeapByteBuffer[pos=0 lim=16 cap=401], payload =
java.nio.HeapByteBuffer[pos=0 lim=381 cap=381]),2835905123),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2641803275, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=1079], payload =
java.nio.HeapByteBuffer[pos=0 lim=1061 cap=1061]),2835905124),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 217385418, key =
java.nio.HeapByteBuffer[pos=0 lim=16 cap=390], payload =
java.nio.HeapByteBuffer[pos=0 lim=370 cap=370]),2835905125),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3321979137, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=398], payload =
java.nio.HeapByteBuffer[pos=0 lim=380 cap=380]),2835905126),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3220348147, key =
java.nio.HeapByteBuffer[pos=0 lim=15 cap=401], payload =
java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905127),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2893466760, key =
java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload =
java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905128),
MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2397577902, key =
java.nio.HeapByteBuffer[pos=0 lim=12 cap=491], payload =
java.nio.HeapByteBuffer[pos=0 lim=475 cap=475]),2835905129)))], HW:
[2835905130], readSize: [50000000], isReadFromLogEnd: [false], error: [None],
[email-events-sox-verified,11] -> Fetch Data: [FetchDataInfo(1149863378
[1149653464 : 125581681],FileMessageSet(MessageAndOffset(Message(magic = 0,
attributes = 2, crc = 3823559139, key = null, payload =
java.nio.HeapByteBuffer[pos=0 lim=1262 cap=1262]),1149863378),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 4139901328, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1224 cap=1224]),1149863379),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 362718848, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1271 cap=1271]),1149863380),
MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3513645936, key =
null, payload = java.nio.HeapByteBuffer[pos=0 lim=1265
cap=1265]),1149863381)))], HW: [1149863327], readSize: [50000000],
isReadFromLogEnd: [false], error: [None], [email-events-sox-verified,3] ->
Fetch Data: [FetchDataInfo(1149891362 [1149636809 :
160852840],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2,
crc = 712083310, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1244
cap=1244]),1149891362), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 1049476577, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1206
cap=1206]),1149891363), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 2736911333, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1199
cap=1199]),1149891364), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 1641412411, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1301
cap=1301]),1149891365), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 2563832660, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1299
cap=1299]),1149891366), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 54053881, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1269
cap=1269]),1149891367), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 3979896138, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1287
cap=1287]),1149891368)))], HW: [1149891315], readSize: [50000000],
isReadFromLogEnd: [false], error: [None]) (kafka.server.ReplicaManager)
[2016-07-28 13:31:36,711] ERROR [Replica Manager on Broker 157]: Error
processing append operation on partition [contacts-pub-sub,7]
(kafka.server.ReplicaManager)
kafka.common.NotEnoughReplicasException: Number of insync replicas for
partition [contacts-pub-sub,7] is [1], below required minimum [2]
[2016-07-28 13:31:36,716] ERROR [Replica Manager on Broker 157]: Error
processing append operation on partition [contacts-pub-sub,7]
(kafka.server.ReplicaManager)
kafka.common.NotEnoughReplicasException: Number of insync replicas for
partition [contacts-pub-sub,7] is [1], below required minimum [2]
[2016-07-28 13:31:36,727] DEBUG [Replica Manager on Broker 157]: Recording
follower broker 160 log read results: Map([biden-rss-events,10] -> Fetch Data:
[FetchDataInfo(2772927 [2692556 : 27087325],ByteBufferMessageSet())], HW:
[2772927], readSize: [50000000], isReadFromLogEnd: [true], error: [None],
[cps-es-indexer-a,28] -> Fetch Data: [FetchDataInfo(519301 [496185 :
68076836],ByteBufferMessageSet())], HW: [519301], readSize: [50000000],
isReadFromLogEnd: [true], error: [None],
[billing-amendment-applied-notifications,7] -> Fetch Data: [FetchDataInfo(2557
[2533 : 2856],ByteBufferMessageSet())], HW: [2557], readSize: [50000000],
isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] -> Fetch Data:
[FetchDataInfo(316637970 [316090773 : 831272496],ByteBufferMessageSet())], HW:
[316637965], readSize: [50000000], isReadFromLogEnd: [true], error: [None],
[contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905130 [2834198288 :
966360310],ByteBufferMessageSet())], HW: [2835905130], readSize: [50000000],
isReadFromLogEnd: [true], error: [None], [email-events-sox-verified,3] -> Fetch
Data: [FetchDataInfo(1149891363 [1149636809 :
160854110],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2,
crc = 1049476577, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1206
cap=1206]),1149891363), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 2736911333, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1199
cap=1199]),1149891364), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 1641412411, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1301
cap=1301]),1149891365), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 2563832660, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1299
cap=1299]),1149891366), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 54053881, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1269
cap=1269]),1149891367), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 3979896138, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1287
cap=1287]),1149891368), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 3636950945, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1303
cap=1303]),1149891369), MessageAndOffset(Message(magic = 0, attributes = 2, crc
= 3954674519, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1210
cap=1210]),1149891370)))], HW: [1149891315], readSize: [50000000],
isReadFromLogEnd: [false], error: [None]) (kafka.server.ReplicaManager)
[2016-07-28 13:31:36,735] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310]
is larger than new hw 2835905130 [2834198288 : 966360310] for partition
[contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 : 966360310]
(kafka.cluster.Partition)
[2016-07-28 13:31:36,736] ERROR [Replica Manager on Broker 157]: Error
processing append operation on partition [contacts-pub-sub,7]
(kafka.server.ReplicaManager)
kafka.common.NotEnoughReplicasException: Number of insync replicas for
partition [contacts-pub-sub,7] is [1], below required minimum [2]
[2016-07-28 13:31:36,736] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Recorded replica 158 log end offset (LEO) position 2835905122 for partition
[contacts-pub-sub,7]. (kafka.cluster.Partition)
[2016-07-28 13:31:36,740] INFO Partition [contacts-pub-sub,7] on broker 157:
Expanding ISR for partition [contacts-pub-sub,7] from 157 to 157,160
(kafka.cluster.Partition)
[2016-07-28 13:31:36,746] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310]
is larger than new hw 2835905130 [2834198288 : 966360310] for partition
[contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 : 966360310]
(kafka.cluster.Partition)
[2016-07-28 13:31:36,746] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Recorded replica 160 log end offset (LEO) position 2835905130 for partition
[contacts-pub-sub,7]. (kafka.cluster.Partition)
[2016-07-28 13:31:36,747] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310]
is larger than new hw 2835905130 [2834198288 : 966360310] for partition
[contacts-pub-sub,7]. All leo's are 2835905131 [2834198288 :
966360747],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
[2016-07-28 13:31:36,750] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310]
is larger than new hw 2835905130 [2834198288 : 966360310] for partition
[contacts-pub-sub,7]. All leo's are 2835905132 [2834198288 :
966361180],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
[2016-07-28 13:31:36,755] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310]
is larger than new hw 2835905130 [2834198288 : 966360310] for partition
[contacts-pub-sub,7]. All leo's are 2835905133 [2834198288 :
966361794],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
[2016-07-28 13:31:36,756] DEBUG Partition [contacts-pub-sub,7] on broker 157:
Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310]
is larger than new hw 2835905130 [2834198288 : 966360310] for partition
[contacts-pub-sub,7]. All leo's are 2835905135 [2834198288 :
966363065],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
