Go to management server (Hope your db is running on same machine) Type the below commands: >mysql cloud > update vm_instance set state="Stopped" where state="Starting";
Delete this vms from CS UI. Hope your agent status is UP and running. Check the nfs moutn points--- Manually mount your nfs and copy some text file and check whether its successful or not. please refer this below link also. Might be useful to you http://www.greenhills.co.uk/2013/08/30/cloudstack-single-server-on-ubuntu-with-kvm.html regards sadhu From: Sugandh S [mailto:[email protected]] Sent: 19 March 2014 14:53 To: Suresh Sadhu; [email protected] Subject: Re: vm stuck in starting state, unable to delete it > You can update the vm state in db as Stopped and try to delete them from CS. How to go about doing this? Sorry, I am very new to CS. This is what I kept on getting when I ran this: grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/management/management-server.log 2014-03-19 13:22:36,156 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744864 to Host 4 timed out after 3600 2014-03-19 13:22:36,156 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:22:36,156 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:23:22,786 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:23:22,786 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489624: Timed out on null 2014-03-19 13:23:22,788 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600 2014-03-19 13:23:22,788 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600 2014-03-19 13:23:36,747 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:23:36,748 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744865: Timed out on null 2014-03-19 13:23:36,748 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744865 to Host 4 timed out after 3600 2014-03-19 13:23:36,748 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics. 2014-03-19 13:23:36,748 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4 2014-03-19 13:24:22,792 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:24:22,792 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489625: Timed out on null 2014-03-19 13:24:22,794 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600 2014-03-19 13:24:22,794 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600 2014-03-19 13:24:37,336 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:24:37,336 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744866: Timed out on null 2014-03-19 13:24:37,336 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744866 to Host 4 timed out after 3600 2014-03-19 13:24:37,336 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 13:24:37,336 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4 2014-03-19 13:25:22,799 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:25:22,799 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489626: Timed out on null 2014-03-19 13:25:22,799 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600 2014-03-19 13:25:22,799 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600 2014-03-19 13:25:38,015 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:25:38,015 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744867: Timed out on null 2014-03-19 13:25:38,015 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744867 to Host 4 timed out after 3600 2014-03-19 13:25:38,015 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 13:25:38,015 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4 2014-03-19 13:26:22,804 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:26:22,804 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489627: Timed out on null 2014-03-19 13:26:22,804 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600 2014-03-19 13:26:22,804 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600 2014-03-19 13:26:38,611 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:26:38,612 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744868: Timed out on null 2014-03-19 13:26:38,612 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744868 to Host 4 timed out after 3600 2014-03-19 13:26:38,612 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:26:38,612 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:27:22,808 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:27:22,809 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489628: Timed out on null 2014-03-19 13:27:22,809 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600 2014-03-19 13:27:22,809 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600 2014-03-19 13:27:39,200 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:27:39,200 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744869: Timed out on null 2014-03-19 13:27:39,200 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744869 to Host 4 timed out after 3600 2014-03-19 13:27:39,200 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 13:27:39,200 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4 2014-03-19 13:28:22,813 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:28:22,813 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489629: Timed out on null 2014-03-19 13:28:22,813 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600 2014-03-19 13:28:22,813 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600 2014-03-19 13:28:39,795 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:28:39,795 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744870: Timed out on null 2014-03-19 13:28:39,796 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744870 to Host 4 timed out after 3600 2014-03-19 13:28:39,796 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:28:39,796 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:29:22,818 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:29:22,818 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489630: Timed out on null 2014-03-19 13:29:22,819 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600 2014-03-19 13:29:22,819 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600 2014-03-19 13:29:40,395 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:29:40,395 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744871: Timed out on null 2014-03-19 13:29:40,396 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744871 to Host 4 timed out after 3600 2014-03-19 13:29:40,396 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:29:40,396 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:30:22,824 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:30:22,824 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489631: Timed out on null 2014-03-19 13:30:22,824 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600 2014-03-19 13:30:22,824 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600 2014-03-19 13:30:40,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:30:40,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744872: Timed out on null 2014-03-19 13:30:40,988 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744872 to Host 4 timed out after 3600 2014-03-19 13:30:40,988 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics. 2014-03-19 13:30:40,988 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4 2014-03-19 13:31:22,828 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:31:22,828 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489632: Timed out on null 2014-03-19 13:31:22,828 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600 2014-03-19 13:31:22,828 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600 2014-03-19 13:31:41,580 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:31:41,580 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744873: Timed out on null 2014-03-19 13:31:41,580 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744873 to Host 4 timed out after 3600 2014-03-19 13:31:41,580 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:31:41,580 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:32:22,833 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:32:22,833 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489633: Timed out on null 2014-03-19 13:32:22,833 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600 2014-03-19 13:32:22,833 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600 2014-03-19 13:32:42,171 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:32:42,172 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744874: Timed out on null 2014-03-19 13:32:42,172 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744874 to Host 4 timed out after 3600 2014-03-19 13:32:42,172 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics. 2014-03-19 13:32:42,172 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4 2014-03-19 13:33:22,838 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:33:22,838 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489634: Timed out on null 2014-03-19 13:33:22,838 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600 2014-03-19 13:33:22,838 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600 2014-03-19 13:33:42,763 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:33:42,763 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744875: Timed out on null 2014-03-19 13:33:42,764 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744875 to Host 4 timed out after 3600 2014-03-19 13:33:42,764 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 13:33:42,764 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4 2014-03-19 13:34:22,843 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:34:22,843 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489635: Timed out on null 2014-03-19 13:34:22,843 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600 2014-03-19 13:34:22,843 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600 2014-03-19 13:34:43,356 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:34:43,356 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744876: Timed out on null 2014-03-19 13:34:43,356 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744876 to Host 4 timed out after 3600 2014-03-19 13:34:43,356 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:34:43,356 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:35:22,847 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:35:22,848 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489636: Timed out on null 2014-03-19 13:35:22,848 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600 2014-03-19 13:35:22,848 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600 2014-03-19 13:35:43,948 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:35:43,948 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744877: Timed out on null 2014-03-19 13:35:43,948 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744877 to Host 4 timed out after 3600 2014-03-19 13:35:43,948 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:35:43,948 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:36:22,852 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:36:22,852 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489637: Timed out on null 2014-03-19 13:36:22,852 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600 2014-03-19 13:36:22,852 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600 2014-03-19 13:36:44,501 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:36:44,501 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744878: Timed out on null 2014-03-19 13:36:44,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744878 to Host 4 timed out after 3600 2014-03-19 13:36:44,502 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 13:36:44,502 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4 2014-03-19 13:37:22,857 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:37:22,857 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489638: Timed out on null 2014-03-19 13:37:22,857 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600 2014-03-19 13:37:22,857 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600 2014-03-19 13:37:45,091 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:37:45,091 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744879: Timed out on null 2014-03-19 13:37:45,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744879 to Host 4 timed out after 3600 2014-03-19 13:37:45,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:37:45,092 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:38:22,862 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:38:22,862 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489639: Timed out on null 2014-03-19 13:38:22,862 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600 2014-03-19 13:38:22,862 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600 2014-03-19 13:38:45,692 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:38:45,692 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744880: Timed out on null 2014-03-19 13:38:45,692 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744880 to Host 4 timed out after 3600 2014-03-19 13:38:45,692 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics. 2014-03-19 13:38:45,692 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4 2014-03-19 13:39:22,866 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:39:22,866 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489640: Timed out on null 2014-03-19 13:39:22,867 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600 2014-03-19 13:39:22,867 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600 2014-03-19 13:39:46,284 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:39:46,284 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744881: Timed out on null 2014-03-19 13:39:46,284 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744881 to Host 4 timed out after 3600 2014-03-19 13:39:46,284 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 13:39:46,284 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4 2014-03-19 13:40:22,871 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:40:22,871 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489641: Timed out on null 2014-03-19 13:40:22,871 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600 2014-03-19 13:40:22,871 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600 2014-03-19 13:40:46,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:40:46,876 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744882: Timed out on null 2014-03-19 13:40:46,876 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744882 to Host 4 timed out after 3600 2014-03-19 13:40:46,876 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 13:40:46,876 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 13:41:22,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:41:22,876 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489642: Timed out on null 2014-03-19 13:41:22,876 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600 2014-03-19 13:41:22,876 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600 2014-03-19 13:41:47,467 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:41:47,468 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744883: Timed out on null 2014-03-19 13:41:47,468 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744883 to Host 4 timed out after 3600 2014-03-19 13:41:47,468 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics. 2014-03-19 13:41:47,468 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4 2014-03-19 13:42:22,881 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:42:22,881 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489643: Timed out on null 2014-03-19 13:42:22,881 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600 2014-03-19 13:42:22,881 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600 2014-03-19 13:42:48,059 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:42:48,060 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744884: Timed out on null 2014-03-19 13:42:48,060 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744884 to Host 4 timed out after 3600 2014-03-19 13:42:48,060 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 13:42:48,060 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4 2014-03-19 13:43:22,886 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:43:22,886 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489644: Timed out on null 2014-03-19 13:43:22,886 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489644 to Host 3 timed out after 3600 2014-03-19 13:43:22,886 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <[email protected]<mailto:[email protected]>> wrote: Can you please provide the logs and also did you notice any exception in the management log. For deleting vm : You can update the vm state in db as Stopped and try to delete them from CS. Regards Sadhu -----Original Message----- From: Sugandh S [mailto:[email protected]<mailto:[email protected]>] Sent: 19 March 2014 14:30 To: [email protected]<mailto:[email protected]> Subject: vm stuck in starting state, unable to delete it Hello all, I am using CS 4.2 and my setup is as follows: One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary. Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent. Now, when I create vms they are stuck in starting state and I am unable to delete them. Any and all help would be greatly appreciated. Thanks ahead, Sugandh
