[
https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187230#comment-15187230
]
Flavio Junqueira commented on KAFKA-3173:
-----------------------------------------
I actually can't reorder it like that, it breaks the controller if I do it, and
in fact, the race condition as I described above can't happen because the
listener callbacks check whether the corresponding state machines have started
or not.
There are two other possible races I found, though.
1- In {{onControllerFailover}}, we have the following:
{noformat}
replicaStateMachine.startup()
partitionStateMachine.startup()
{noformat}
By invoking {{replicaStateMachine.startup()}} first, we set {{hasStarted}} to
true. If there is a broker change, then the listener callback will be executed
({{ReplicaStateMahcine.hasSarted}} is true), which will eventually invoke
{{partitionStateMachine.triggerOnlinePartitionStateChange()}}. If
{{partitionStateMachine.startup()}} hasn't started yet, then
{{partitionStateMachine.hasStarted}} is false and we will end up with a dirty
batch. We need to swap those calls. I also checked that there is no flow the
other way around (partitionStateMachine -> replicaStateMachine), so it is safe
to swap the calls.
2- If a zookeeper event is triggered while either state machine is starting up,
then it is possible that either the startup call or the handling of the
zookeeper event will find a dirty batch because of concurrent execution. In
this case, we don't leave a dirty batch because it clears after the execution,
but one of the calls might fail and we don't have code to recover.
We are currently locking many of these calls with a controller lock to prevent
races, but apparently we forgot to add it to the startup calls, like this:
{noformat}
inLock(controllerContext.controllerLock) {
triggerOnlinePartitionStateChange()
}
{noformat}
I verified that this doesn't break anything, but I want to write a test case to
see if I can repro the problem.
> Error while moving some partitions to OnlinePartition state
> ------------------------------------------------------------
>
> Key: KAFKA-3173
> URL: https://issues.apache.org/jira/browse/KAFKA-3173
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.0
> Reporter: Flavio Junqueira
> Assignee: Flavio Junqueira
> Priority: Critical
> Fix For: 0.10.0.0
>
>
> We observed another instance of the problem reported in KAFKA-2300, but this
> time the error appeared in the partition state machine. In KAFKA-2300, we
> haven't cleaned up the state in {{PartitionStateMachine}} and
> {{ReplicaStateMachine}} as we do in {{KafkaController}}.
> Here is the stack trace:
> {noformat}
> 2016-01-29 15:26:51,393] ERROR [Partition state machine on Controller 0]:
> Error while moving some partitions to OnlinePartition state
> (kafka.controller.PartitionStateMachine)java.lang.IllegalStateException:
> Controller to broker state change requests batch is not empty while creating
> a new one.
> Some LeaderAndIsr state changes Map(0 -> Map(foo-0 -> (LeaderAndIsrInfo:
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0)))
> might be lost at
> kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:254)
> at
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:144)
> at
> kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:517)
> at
> kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:504)
> at
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(PartitionStateMachine.scala:437)
> at
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at
> kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262) at
> kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:418)
> at
> org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842) at
> org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
