[
https://issues.apache.org/jira/browse/KAFKA-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KAFKA-6747:
--------------------------
Description:
[~frederica] running tests against kafka-streams 1.1 and get the following
stack trace (everything was working alright using kafka-streams 1.0):
{code}
ERROR org.apache.kafka.streams.processor.internals.AssignedStreamsTasks -
stream-thread [feedBuilder-XXX-StreamThread-4] Failed to close stream task, 0_2
org.apache.kafka.common.KafkaException: TransactionalId feedBuilder-0_2:
Invalid transition attempted from state READY to state ABORTING_TRANSACTION
at
org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:757)
at
org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:751)
at
org.apache.kafka.clients.producer.internals.TransactionManager.beginAbort(TransactionManager.java:230)
at
org.apache.kafka.clients.producer.KafkaProducer.abortTransaction(KafkaProducer.java:660)
at
org.apache.kafka.streams.processor.internals.StreamTask.closeSuspended(StreamTask.java:486)
at
org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:546)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.closeNonRunningTasks(AssignedTasks.java:166)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.suspend(AssignedTasks.java:151)
at
org.apache.kafka.streams.processor.internals.TaskManager.suspendTasksAndState(TaskManager.java:242)
at
org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsRevoked(StreamThread.java:291)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:414)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:359)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
at
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:827)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:784)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)
{code}
This happens when starting the same stream-processing application on 3 JVMs all
running on the same linux box, JVMs are named JVM-[2-4]. All 3 instances use
separate stream state.dir. No record is ever processed because the input kafka
topics are empty at this stage.
JVM-2 starts first, joined shortly after by JVM-4 and JVM-3, find the state
transition logs below. The above stacktrace is from JVM-4
{code}
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
JVM-4 crashes here with above stacktrace
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
ERROR
[JVM-4] stream-client [feedBuilder-XXX] State transition from ERROR to
PENDING_SHUTDOWN
[JVM-4] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
PENDING_SHUTDOWN
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
PENDING_SHUTDOWN
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
PENDING_SHUTDOWN
[JVM-3] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
{code}
was:
I running tests against kafka-streams 1.1 and get the following stack trace
(everything was working alright using kafka-streams 1.0):
{code}
ERROR org.apache.kafka.streams.processor.internals.AssignedStreamsTasks -
stream-thread [feedBuilder-XXX-StreamThread-4] Failed to close stream task, 0_2
org.apache.kafka.common.KafkaException: TransactionalId feedBuilder-0_2:
Invalid transition attempted from state READY to state ABORTING_TRANSACTION
at
org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:757)
at
org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:751)
at
org.apache.kafka.clients.producer.internals.TransactionManager.beginAbort(TransactionManager.java:230)
at
org.apache.kafka.clients.producer.KafkaProducer.abortTransaction(KafkaProducer.java:660)
at
org.apache.kafka.streams.processor.internals.StreamTask.closeSuspended(StreamTask.java:486)
at
org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:546)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.closeNonRunningTasks(AssignedTasks.java:166)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.suspend(AssignedTasks.java:151)
at
org.apache.kafka.streams.processor.internals.TaskManager.suspendTasksAndState(TaskManager.java:242)
at
org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsRevoked(StreamThread.java:291)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:414)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:359)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
at
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:827)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:784)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)
{code}
This happens when starting the same stream-processing application on 3 JVMs all
running on the same linux box, JVMs are named JVM-[2-4]. All 3 instances use
separate stream state.dir. No record is ever processed because the input kafka
topics are empty at this stage.
JVM-2 starts first, joined shortly after by JVM-4 and JVM-3, find the state
transition logs below. The above stacktrace is from JVM-4
{code}
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
JVM-4 crashes here with above stacktrace
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
ERROR
[JVM-4] stream-client [feedBuilder-XXX] State transition from ERROR to
PENDING_SHUTDOWN
[JVM-4] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
PENDING_SHUTDOWN
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
REBALANCING
[JVM-2] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
RUNNING
[JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
PENDING_SHUTDOWN
[JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
PENDING_SHUTDOWN
[JVM-3] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
[JVM-4] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN
to NOT_RUNNING
{code}
> kafka-streams Invalid transition attempted from state READY to state
> ABORTING_TRANSACTION
> -----------------------------------------------------------------------------------------
>
> Key: KAFKA-6747
> URL: https://issues.apache.org/jira/browse/KAFKA-6747
> Project: Kafka
> Issue Type: Bug
> Reporter: Ted Yu
> Priority: Major
>
> [~frederica] running tests against kafka-streams 1.1 and get the following
> stack trace (everything was working alright using kafka-streams 1.0):
> {code}
> ERROR org.apache.kafka.streams.processor.internals.AssignedStreamsTasks -
> stream-thread [feedBuilder-XXX-StreamThread-4] Failed to close stream task,
> 0_2
> org.apache.kafka.common.KafkaException: TransactionalId feedBuilder-0_2:
> Invalid transition attempted from state READY to state ABORTING_TRANSACTION
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:757)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:751)
> at
> org.apache.kafka.clients.producer.internals.TransactionManager.beginAbort(TransactionManager.java:230)
> at
> org.apache.kafka.clients.producer.KafkaProducer.abortTransaction(KafkaProducer.java:660)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.closeSuspended(StreamTask.java:486)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:546)
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.closeNonRunningTasks(AssignedTasks.java:166)
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.suspend(AssignedTasks.java:151)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.suspendTasksAndState(TaskManager.java:242)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsRevoked(StreamThread.java:291)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:414)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:359)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:827)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:784)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)
> {code}
> This happens when starting the same stream-processing application on 3 JVMs
> all running on the same linux box, JVMs are named JVM-[2-4]. All 3 instances
> use separate stream state.dir. No record is ever processed because the input
> kafka topics are empty at this stage.
> JVM-2 starts first, joined shortly after by JVM-4 and JVM-3, find the state
> transition logs below. The above stacktrace is from JVM-4
> {code}
> [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> JVM-4 crashes here with above stacktrace
> [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> ERROR
> [JVM-4] stream-client [feedBuilder-XXX] State transition from ERROR to
> PENDING_SHUTDOWN
> [JVM-4] stream-client [feedBuilder-XXX] State transition from
> PENDING_SHUTDOWN to NOT_RUNNING
> [JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to
> PENDING_SHUTDOWN
> [JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
> REBALANCING
> [JVM-2] stream-client [feedBuilder-XXX] State transition from
> PENDING_SHUTDOWN to NOT_RUNNING
> [JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to
> RUNNING
> [JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to
> PENDING_SHUTDOWN
> [JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to
> PENDING_SHUTDOWN
> [JVM-3] stream-client [feedBuilder-XXX] State transition from
> PENDING_SHUTDOWN to NOT_RUNNING
> [JVM-4] stream-client [feedBuilder-XXX] State transition from
> PENDING_SHUTDOWN to NOT_RUNNING
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
