Nilotpal Nandi created HDDS-1126:
------------------------------------
Summary: datanode is trying to qausi-close a container which is
already closed
Key: HDDS-1126
URL: https://issues.apache.org/jira/browse/HDDS-1126
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Nilotpal Nandi
steps taken :
--------------------
# created 12 datanodes cluster and running workload on all the nodes
# running failure injection/restart on 1 datanode at a time periodically and
randomly.
Error seen in ozone.log :
--------------------------
{noformat}
2019-02-18 06:06:32,780 [Datanode State Machine Thread - 0] DEBUG
(DatanodeStateMachine.java:176) - Executing cycle Number : 30
2019-02-18 06:06:32,784 [Command processor thread] DEBUG
(CloseContainerCommandHandler.java:71) - Processing Close Container command.
2019-02-18 06:06:32,785 [Datanode State Machine Thread - 0] DEBUG
(DatanodeStateMachine.java:176) - Executing cycle Number : 31
2019-02-18 06:06:32,785 [Command processor thread] ERROR
(CloseContainerCommandHandler.java:118) - Can't close container #37
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Cannot quasi close container #37 while in CLOSED state.
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.quasiCloseContainer(KeyValueHandler.java:903)
at
org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.quasiCloseContainer(ContainerController.java:93)
at
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:110)
at
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93)
at
org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:413)
at java.lang.Thread.run(Thread.java:748)
2019-02-18 06:06:32,785 [Command processor thread] DEBUG
(CloseContainerCommandHandler.java:71) - Processing Close Container command.
2019-02-18 06:06:32,788 [Command processor thread] DEBUG
(CloseContainerCommandHandler.java:71) - Processing Close Container command.
2019-02-18 06:06:32,788 [Datanode State Machine Thread - 0] DEBUG
(DatanodeStateMachine.java:176) - Executing cycle Number : 32
2019-02-18 06:06:34,430 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:36,608 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:38,876 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:41,084 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:43,297 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:45,469 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:47,684 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:49,958 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:52,124 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:54,344 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:56,499 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:06:58,764 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:07:00,969 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:07:02,788 [Datanode State Machine Thread - 0] DEBUG
(DatanodeStateMachine.java:176) - Executing cycle Number : 33
2019-02-18 06:07:03,240 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
2019-02-18 06:07:05,486 [main] DEBUG (OzoneClientFactory.java:287) - Using
org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
