[
https://issues.apache.org/jira/browse/KAFKA-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021577#comment-16021577
]
Abhisek Saikia edited comment on KAFKA-5296 at 5/23/17 6:45 PM:
----------------------------------------------------------------
Any new topic creation is running into issues now. Here are the steps I am
using to reproduce the issue-
1. Create a new topic with 20 partitions and 3 replicas -
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3
--partitions 20 --topic testcreate2
2. Get the topic details using kafka-topics command-
kafka-topics.sh --describe --zookeeper localhost:2181 --topic testcreate2
Topic:testcreate2 PartitionCount:20 ReplicationFactor:3 Configs:
Topic: testcreate2 Partition: 0 Leader: 138 Replicas:
138,132,133 Isr: 138,132,133
Topic: testcreate2 Partition: 1 Leader: 139 Replicas:
139,133,135 Isr: 139,133,135
Topic: testcreate2 Partition: 2 Leader: 140 Replicas:
140,135,136 Isr: 140,135,136
Topic: testcreate2 Partition: 3 Leader: 130 Replicas:
130,136,137 Isr: 130,136,137
Topic: testcreate2 Partition: 4 Leader: 131 Replicas:
131,137,138 Isr: 131,137,138
Topic: testcreate2 Partition: 5 Leader: 132 Replicas:
132,138,139 Isr: 132,138,139
Topic: testcreate2 Partition: 6 Leader: 133 Replicas:
133,139,140 Isr: 133,139,140
Topic: testcreate2 Partition: 7 Leader: 135 Replicas:
135,140,130 Isr: 135,140,130
Topic: testcreate2 Partition: 8 Leader: 136 Replicas:
136,130,131 Isr: 136,130,131
Topic: testcreate2 Partition: 9 Leader: 137 Replicas:
137,131,132 Isr: 137,131,132
Topic: testcreate2 Partition: 10 Leader: 138 Replicas:
138,133,135 Isr: 138,133,135
Topic: testcreate2 Partition: 11 Leader: 139 Replicas:
139,135,136 Isr: 139,135,136
Topic: testcreate2 Partition: 12 Leader: 140 Replicas:
140,136,137 Isr: 140,136,137
Topic: testcreate2 Partition: 13 Leader: 130 Replicas:
130,137,138 Isr: 130,137,138
Topic: testcreate2 Partition: 14 Leader: 131 Replicas:
131,138,139 Isr: 131,138,139
Topic: testcreate2 Partition: 15 Leader: 132 Replicas:
132,139,140 Isr: 132,139,140
Topic: testcreate2 Partition: 16 Leader: 133 Replicas:
133,140,130 Isr: 133,140,130
Topic: testcreate2 Partition: 17 Leader: 135 Replicas:
135,130,131 Isr: 135,130,131
Topic: testcreate2 Partition: 18 Leader: 136 Replicas:
136,131,132 Isr: 136,131,132
Topic: testcreate2 Partition: 19 Leader: 137 Replicas:
137,132,133 Isr: 137,132,133
3. Using Simple consumer try to read the topic metadata for this topic. It
seems topic metadata is found for only 11 partitions out of 20.
{
"messages": [
{
"topicMetaDataList": [
{
"topic": "testcreate2",
"paritionMetaData": [
{
"partition": 0,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 132,
"port": 9092
},
{
"id": 133,
"port": 9092
}
]
},
{
"partition": 1,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 18,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 3,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 136,
"port": 9092
},
{
"id": 137,
"port": 9092
}
]
},
{
"partition": 4,
"leader": {
"id": 131,
"port": 9092
},
"isr": [
{
"id": 131,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
},
{
"partition": 5,
"leader": {
"id": 132,
"port": 9092
},
"isr": [
{
"id": 132,
"port": 9092
},
{
"id": 138,
"port": 9092
},
{
"id": 139,
"port": 9092
}
]
},
{
"partition": 8,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 130,
"port": 9092
},
{
"id": 131,
"port": 9092
}
]
},
{
"partition": 9,
"leader": {
"id": 137,
"port": 9092
},
"isr": [
{
"id": 137,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 10,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 11,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 135,
"port": 9092
},
{
"id": 136,
"port": 9092
}
]
},
{
"partition": 13,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
}
]
}
]
}
]
}
4. Above payload does not contain metadata for partition 14 but as per
kafka-topics.sh command , broker 131 is supposed to be the leader broker for
this partition.
The log directory of this broker does not have any folder with name
"testcreate2-2" . It has the following folders -
testcreate2-18:
00000000000000000000.index 00000000000000000000.log
00000000000000000000.timeindex
testcreate2-4:
00000000000000000000.index 00000000000000000000.log
00000000000000000000.timeindex
testcreate2-8:
00000000000000000000.index 00000000000000000000.log
00000000000000000000.timeindex
testcreate2-9:
00000000000000000000.index 00000000000000000000.log
00000000000000000000.timeindex
As per the following logs of broker 131, its receiving LeaderAndIsr request
from multiple controller for different paritions.
./state-change.log.2017-05-22-18:[2017-05-22 18:48:30,108] TRACE Broker 131
received LeaderAndIsr request PartitionState(controllerEpoch=64, leader=131,
leaderEpoch=0, isr=[131, 138, 139], zkVersion=0, replicas=[139, 131, 138])
correlation id 5453 from controller 130 epoch 64 for partition [testcreate2,14]
(state.change.logger)
./state-change.log.2017-05-22-18:[2017-05-22 18:48:30,106] TRACE Broker 131
received LeaderAndIsr request PartitionState(controllerEpoch=70, leader=136,
leaderEpoch=0, isr=[136, 131, 132], zkVersion=0, replicas=[132, 136, 131])
correlation id 814 from controller 137 epoch 70 for partition [testcreate2,18]
(state.change.logger)
was (Author: asaikia):
Any new topic creation is running into issues now. Here are the steps I am
using to reproduce the issue-
1. Create a new topic with 20 partitions and 3 replicas -
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3
--partitions 20 --topic testcreate2
2. Get the topic details using kafka-topics command-
kafka-topics.sh --describe --zookeeper localhost:2181 --topic testcreate2
Topic:testcreate2 PartitionCount:20 ReplicationFactor:3 Configs:
Topic: testcreate2 Partition: 0 Leader: 138 Replicas:
138,132,133 Isr: 138,132,133
Topic: testcreate2 Partition: 1 Leader: 139 Replicas:
139,133,135 Isr: 139,133,135
Topic: testcreate2 Partition: 2 Leader: 140 Replicas:
140,135,136 Isr: 140,135,136
Topic: testcreate2 Partition: 3 Leader: 130 Replicas:
130,136,137 Isr: 130,136,137
Topic: testcreate2 Partition: 4 Leader: 131 Replicas:
131,137,138 Isr: 131,137,138
Topic: testcreate2 Partition: 5 Leader: 132 Replicas:
132,138,139 Isr: 132,138,139
Topic: testcreate2 Partition: 6 Leader: 133 Replicas:
133,139,140 Isr: 133,139,140
Topic: testcreate2 Partition: 7 Leader: 135 Replicas:
135,140,130 Isr: 135,140,130
Topic: testcreate2 Partition: 8 Leader: 136 Replicas:
136,130,131 Isr: 136,130,131
Topic: testcreate2 Partition: 9 Leader: 137 Replicas:
137,131,132 Isr: 137,131,132
Topic: testcreate2 Partition: 10 Leader: 138 Replicas:
138,133,135 Isr: 138,133,135
Topic: testcreate2 Partition: 11 Leader: 139 Replicas:
139,135,136 Isr: 139,135,136
Topic: testcreate2 Partition: 12 Leader: 140 Replicas:
140,136,137 Isr: 140,136,137
Topic: testcreate2 Partition: 13 Leader: 130 Replicas:
130,137,138 Isr: 130,137,138
Topic: testcreate2 Partition: 14 Leader: 131 Replicas:
131,138,139 Isr: 131,138,139
Topic: testcreate2 Partition: 15 Leader: 132 Replicas:
132,139,140 Isr: 132,139,140
Topic: testcreate2 Partition: 16 Leader: 133 Replicas:
133,140,130 Isr: 133,140,130
Topic: testcreate2 Partition: 17 Leader: 135 Replicas:
135,130,131 Isr: 135,130,131
Topic: testcreate2 Partition: 18 Leader: 136 Replicas:
136,131,132 Isr: 136,131,132
Topic: testcreate2 Partition: 19 Leader: 137 Replicas:
137,132,133 Isr: 137,132,133
3. Using Simple consumer try to read the topic metadata for this topic. It
seems topic metadata is found for only 11 partitions out of 20.
{
"messages": [
{
"topicMetaDataList": [
{
"topic": "testcreate2",
"paritionMetaData": [
{
"partition": 0,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 132,
"port": 9092
},
{
"id": 133,
"port": 9092
}
]
},
{
"partition": 1,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 18,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 3,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 136,
"port": 9092
},
{
"id": 137,
"port": 9092
}
]
},
{
"partition": 4,
"leader": {
"id": 131,
"port": 9092
},
"isr": [
{
"id": 131,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
},
{
"partition": 5,
"leader": {
"id": 132,
"port": 9092
},
"isr": [
{
"id": 132,
"port": 9092
},
{
"id": 138,
"port": 9092
},
{
"id": 139,
"port": 9092
}
]
},
{
"partition": 8,
"leader": {
"id": 136,
"port": 9092
},
"isr": [
{
"id": 136,
"port": 9092
},
{
"id": 130,
"port": 9092
},
{
"id": 131,
"port": 9092
}
]
},
{
"partition": 9,
"leader": {
"id": 137,
"port": 9092
},
"isr": [
{
"id": 137,
"port": 9092
},
{
"id": 131,
"port": 9092
},
{
"id": 132,
"port": 9092
}
]
},
{
"partition": 10,
"leader": {
"id": 138,
"port": 9092
},
"isr": [
{
"id": 138,
"port": 9092
},
{
"id": 133,
"port": 9092
},
{
"id": 135,
"port": 9092
}
]
},
{
"partition": 11,
"leader": {
"id": 139,
"port": 9092
},
"isr": [
{
"id": 139,
"port": 9092
},
{
"id": 135,
"port": 9092
},
{
"id": 136,
"port": 9092
}
]
},
{
"partition": 13,
"leader": {
"id": 130,
"port": 9092
},
"isr": [
{
"id": 130,
"port": 9092
},
{
"id": 137,
"port": 9092
},
{
"id": 138,
"port": 9092
}
]
}
]
}
]
}
]
}
4. Above payload does not contain metadata for partition 2 but as per
kafka-topics.sh command , broker 140 is supposed to be the leader broker for
this partition.
The log directory of this broker does not have any folder with name
"testcreate2-2"
> Unable to write to some partitions of newly created topic in 10.2
> -----------------------------------------------------------------
>
> Key: KAFKA-5296
> URL: https://issues.apache.org/jira/browse/KAFKA-5296
> Project: Kafka
> Issue Type: Bug
> Reporter: Abhisek Saikia
>
> We are using kafka 10.2 and the cluster was running fine for a month with 50
> topics and now we are having issue in producing message by creating new
> topics. The create topic command is successful but producers are throwing
> error while writing to some partitions.
> Error in producer-
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for
> [topic1]-8: 30039 ms has passed since batch creation plus linger time
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70)
> ~[kafka-clients-0.10.2.0.jar:na]
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57)
> ~[kafka-clients-0.10.2.0.jar:na]
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
> ~[kafka-clients-0.10.2.0.jar:na]
> On the broker side, I don't see any topic-parition folder getting created for
> the broker who is the leader for the partition.
> While using 0.8 client, the write used to hang while it starts writing to the
> partition having this issue. With 10.2 it resolved the the producer hang issue
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
