Boyang Chen created KAFKA-9674:
----------------------------------
Summary: Task corruption should also close the producer if
necessary
Key: KAFKA-9674
URL: https://issues.apache.org/jira/browse/KAFKA-9674
Project: Kafka
Issue Type: Bug
Reporter: Boyang Chen
Assignee: Boyang Chen
The task revive call only transits the task to CREATED mode. It should handle
the recreation of task producer as well.
Sequence is like:
# Task hits out of range exception and throws CorruptedException
# Task producer closed along with the task
# Task revived and rebalance triggered
# Task was assigned back to the same thread
# Trying to use task producer will throw as it has already been closed.
The full log:
[2020-03-03T21:56:29-08:00]
(streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04
05:56:29,070] WARN
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3]
stream-thread
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3]
Encountered org.apache.kafka.clients.consumer.OffsetOutOfRangeException
fetching records from restore consumer for partitions
[stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000019-changelog-0], it is
likely that the consumer's position has fallen out of the topic partition
offset range because the topic was truncated or compacted on the broker,
marking the corresponding tasks as corrupted and re-initializing it later.
(org.apache.kafka.streams.processor.internals.StoreChangelogReader)
[2020-03-03T21:56:29-08:00]
(streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04
05:56:29,071] WARN
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3]
stream-thread
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] Detected
the states of tasks
\{1_0=[stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000019-changelog-0]}
are corrupted. Will close the task as dirty and re-create and bootstrap from
scratch. (org.apache.kafka.streams.processor.internals.StreamThread)
[2020-03-03T21:56:30-08:00]
(streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04
05:56:30,010] INFO
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3]
[Producer
clientId=stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3-1_0-producer,
transactionalId=stream-soak-test-1_0] Closing the Kafka producer with
timeoutMillis = 9223372036854775807 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2020-03-03T21:56:30-08:00]
(streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04
05:56:30,017] INFO
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3]
stream-thread
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] task
[1_0] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask)
[2020-03-03T21:56:22-08:00]
(streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04
05:56:22,827] INFO
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3]
[Producer
clientId=stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3-1_0-producer,
transactionalId=stream-soak-test-1_0] Closing the Kafka producer with
timeoutMillis = 9223372036854775807 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2020-03-03T21:56:22-08:00]
(streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04
05:56:22,829] INFO
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3]
stream-thread
[stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] task
[1_0] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
