[
https://issues.apache.org/jira/browse/KAFKA-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712090#comment-17712090
]
Justine Olshan commented on KAFKA-14904:
----------------------------------------
Jenkins truncates this out since the bounced broker leads to a ton of aborted
transaction errors, but after that we see:
{code:java}
[2023-04-13 14:15:44,487] WARN [Producer clientId=producer-test-txn,
transactionalId=test-txn] Got error produce response with correlation id 16 on
topic-partition output-topic-0, retrying (2147483646 attempts left). Error:
CONCURRENT_TRANSACTIONS (org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:15:44,615] ERROR [ReplicaManager broker=1] Error processing
append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order
sequence number for producer 1 at offset 11 in partition output-topic-0: 0
(incoming seq. number), 21 (current end sequence number)
[2023-04-13 14:15:44,615] WARN [Producer clientId=producer-test-txn,
transactionalId=test-txn] Got error produce response with correlation id 20 on
topic-partition output-topic-0, retrying (2147483645 attempts left). Error:
OUT_OF_ORDER_SEQUENCE_NUMBER
(org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:15:44,686] WARN [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Partition output-topic-0 marked as failed
(kafka.server.ReplicaFetcherThread:70)
{code}
And a TON more out of sequence errors (probably most of the partitions). Will
investigate further if this is a result of my change and if there are ways to
fix this.
Other logs:
{code:java}
[2023-04-13 14:16:49,648] ERROR [ReplicaManager broker=1] Error processing
append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order
sequence number for producer 1 at offset 489 in partition output-topic-0: 0
(incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:49,649] WARN [Producer clientId=producer-test-txn,
transactionalId=test-txn] Got error produce response with correlation id 774 on
topic-partition output-topic-0, retrying (2147483019 attempts left). Error:
OUT_OF_ORDER_SEQUENCE_NUMBER
(org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:49,749] ERROR [ReplicaManager broker=1] Error processing
append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order
sequence number for producer 1 at offset 489 in partition output-topic-0: 0
(incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:49,750] WARN [Producer clientId=producer-test-txn,
transactionalId=test-txn] Got error produce response with correlation id 775 on
topic-partition output-topic-0, retrying (2147483018 attempts left). Error:
OUT_OF_ORDER_SEQUENCE_NUMBER
(org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:49,851] ERROR [ReplicaManager broker=1] Error processing
append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order
sequence number for producer 1 at offset 489 in partition output-topic-0: 0
(incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:49,851] WARN [Producer clientId=producer-test-txn,
transactionalId=test-txn] Got error produce response with correlation id 776 on
topic-partition output-topic-0, retrying (2147483017 attempts left). Error:
OUT_OF_ORDER_SEQUENCE_NUMBER
(org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:49,952] ERROR [ReplicaManager broker=1] Error processing
append operation on partition output-topic-0
(kafka.server.ReplicaManager:76){code}
And
{code:java}
[2023-04-13 14:16:51,573] ERROR [ReplicaManager broker=1] Error processing
append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order
sequence number for producer 1 at offset 489 in partition output-topic-0: 0
(incoming seq. number), 499 (current end sequence number) [2023-04-13
14:16:51,573] WARN [Producer clientId=producer-test-txn,
transactionalId=test-txn] Got error produce response with correlation id 793 on
topic-partition output-topic-0, retrying (2147483000 attempts left). Error:
OUT_OF_ORDER_SEQUENCE_NUMBER
(org.apache.kafka.clients.producer.internals.Sender:630) [2023-04-13
14:16:51,622] ERROR Error when sending message to topic output-topic with key:
1, value: 1 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 5, value: 5 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 7, value: 7 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 8, value: 8 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 11, value: 11 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 15, value: 15 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 17, value: 17 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 21, value: 21 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 22, value: 22 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 23, value: 23 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR
Error when sending message to topic output-topic with key: 25, value: 25 with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully. at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at
java.base/java.lang.Thread.run(Thread.java:833)[2023-04-13 14:16:51,573] ERROR
[ReplicaManager broker=1] Error processing append operation on partition
output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order
sequence number for producer 1 at offset 489 in partition output-topic-0: 0
(incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:51,573] WARN [Producer clientId=producer-test-txn,
transactionalId=test-txn] Got error produce response with correlation id 793 on
topic-partition output-topic-0, retrying (2147483000 attempts left). Error:
OUT_OF_ORDER_SEQUENCE_NUMBER
(org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 1, value: 1 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 5, value: 5 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 7, value: 7 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 8, value: 8 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 11, value: 11 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 15, value: 15 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 17, value: 17 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 21, value: 21 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 22, value: 22 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 23, value: 23 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic
output-topic with key: 25, value: 25 with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833){code}
> Flaky Test kafka.api.TransactionsBounceTest.testWithGroupId()
> --------------------------------------------------------------
>
> Key: KAFKA-14904
> URL: https://issues.apache.org/jira/browse/KAFKA-14904
> Project: Kafka
> Issue Type: Test
> Reporter: Justine Olshan
> Assignee: Justine Olshan
> Priority: Major
>
> After merging KAFKA-14561 I noticed this test still occasionally failed via
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 60000ms while awaiting EndTxn(true)
> I will investigate the cause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
