2019-12-01 03:09:10,715 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-79d54dc9) (logid:b94602f7) Monitor StoragePoolMonitor says there is an error in the connect process for 4 due to Unable to connect host 4 to storage pool id 1 due to com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 1 due to ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat file entry on SR 1562e840-29e3-ba00-e148-6dad6353042a due to #3# Unable to find the host uuid: 0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 host:0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 pool: 172.29.1.97/nfs/export/primary1
Probably NFS permission issue or similar. Try manually mounting NFS on XS to verify connectivity and try writing a file to mount point. Sun, 1 Dec 2019, 05:32 pradeep pal, <[email protected]> wrote: > Hi, > > Ok, let me check. > > Thanks > Pradeep > > > Get Outlook for iOS<https://aka.ms/o0ukef> > ________________________________ > From: li jerry <[email protected]> > Sent: Sunday, December 1, 2019 7:55:18 AM > To: [email protected] <[email protected]> > Subject: 回复: Unable to start a VM due to insufficient capacity > > Please check the connection between your XenServer and NFS primary storage > > > -----邮件原件----- > 发件人: pradeep pal <[email protected]> > 发送时间: 2019年12月1日 7:23 > 收件人: [email protected] > 主题: Unable to start a VM due to insufficient capacity > > > ________________________________ > Hi, > > I am new for the CloudStack. > > > I have installed and configure ACS 4.11.3.0+mysql5.6+XenServer 7.1.CU2 in > my office. > > But when we launch instance I always get "Unable to start a VM due to > insufficient capacity" Can anyone help me to solve this issue. > > Please find the attached log. > > > 2019-12-01 02:58:13,483 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-4:ctx-a1580e0a job-90) (logid:ede69855) Unexpected > exception while executing > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin > > com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due > to insufficient capacity > > Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable > to create a deployment for VM[User|i-2-13-VM]Scope=interface > com.cloud.dc.DataCenter; id=1 > > 2019-12-01 02:58:13,486 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-4:ctx-a1580e0a job-90) (logid:ede69855) Complete async > job-90, jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable > to start a VM due to insufficient capacity"} > > 2019-12-01 03:07:17,393 ERROR [c.c.c.ClusterServiceServletContainer] > (Thread-20:null) (logid:) Unexpected exception > > 2019-12-01 03:08:29,357 WARN [n.s.e.c.ConfigurationFactory] (main:null) > (logid:) No configuration found. Configuring ehcache from > ehcache-failsafe.xml found in the classpath: > jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/ehcache-failsafe.xml > > 2019-12-01 03:08:50,346 INFO [c.c.h.x.r.XenServerConnectionPool] > (main:null) (logid:) XenServer Connection Pool Configs: > sleep.interval.on.error=10000 > > 2019-12-01 03:08:59,000 ERROR [c.c.u.s.SshHelper] > (AgentTaskPool-1:ctx-55835787) (logid:58f334e3) SSH execution of command xe > sm-list | grep "resigning of duplicates" has an error status code in > return. Result output: > > 2019-12-01 03:08:59,124 ERROR [c.c.u.s.SshHelper] > (AgentTaskPool-2:ctx-79d54dc9) (logid:b94602f7) SSH execution of command xe > sm-list | grep "resigning of duplicates" has an error status code in > return. Result output: > > com.cloud.utils.exception.CloudRuntimeException: Unable to setup heartbeat > file entry on SR 1562e840-29e3-ba00-e148-6dad6353042a due to #3# Unable to > find the host uuid: 0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 > > 2019-12-01 03:09:10,715 INFO [c.c.u.e.CSExceptionErrorCode] > (AgentTaskPool-2:ctx-79d54dc9) (logid:b94602f7) Could not find exception: > com.cloud.exception.ConnectionException in error code list for exceptions > > 2019-12-01 03:09:10,715 WARN [c.c.a.m.AgentManagerImpl] > (AgentTaskPool-2:ctx-79d54dc9) (logid:b94602f7) Monitor StoragePoolMonitor > says there is an error in the connect process for 4 due to Unable to > connect host 4 to storage pool id 1 due to > com.cloud.utils.exception.CloudRuntimeException: Unable establish > connection from storage head to storage pool 1 due to > ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat file > entry on SR 1562e840-29e3-ba00-e148-6dad6353042a due to #3# Unable to find > the host uuid: 0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 > host:0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 pool: > 172.29.1.97/nfs/export/primary1 > > com.cloud.exception.ConnectionException: Unable to connect host 4 to > storage pool id 1 due to com.cloud.utils.exception.CloudRuntimeException: > Unable establish connection from storage head to storage pool 1 due to > ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat file > entry on SR 1562e840-29e3-ba00-e148-6dad6353042a due to #3# Unable to find > the host uuid: 0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 > host:0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 pool: > 172.29.1.97/nfs/export/primary1 > > Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable > establish connection from storage head to storage pool 1 due to > ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat file > entry on SR 1562e840-29e3-ba00-e148-6dad6353042a due to #3# Unable to find > the host uuid: 0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 > host:0e9f60c3-cfb7-4038-b00f-319d7e22d7f6 pool: > 172.29.1.97/nfs/export/primary1 > > 2019-12-01 03:10:27,667 ERROR [c.c.u.s.SshHelper] > (AgentTaskPool-3:ctx-f701d1f9) (logid:ceb4daae) SSH execution of command xe > sm-list | grep "resigning of duplicates" has an error status code in > return. Result output: > > 2019-12-01 03:23:49,476 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-1:ctx-f035b3f6 job-99/job-100 ctx-1ee73f54) > (logid:0eb71d41) Cleaning up because we're unable to implement the network > Ntwk[206|Guest|11] > > com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to > allocate vnet as a part of network Ntwk[206|Guest|11] implement > Scope=interface com.cloud.dc.DataCenter; id=1 > > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > > 2019-12-01 03:23:50,229 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-1:ctx-f035b3f6 job-99/job-100 ctx-1ee73f54) > (logid:0eb71d41) Invocation exception, caused by: > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-14-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:23:50,229 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-1:ctx-f035b3f6 job-99/job-100 ctx-1ee73f54) > (logid:0eb71d41) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-14-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM[User|i-2-14-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:23:51,612 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-2:ctx-219e47df job-99/job-101 ctx-74b313f0) > (logid:0eb71d41) Cleaning up because we're unable to implement the network > Ntwk[206|Guest|11] > > com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to > allocate vnet as a part of network Ntwk[206|Guest|11] implement > Scope=interface com.cloud.dc.DataCenter; id=1 > > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > > 2019-12-01 03:23:52,142 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-2:ctx-219e47df job-99/job-101 ctx-74b313f0) > (logid:0eb71d41) Invocation exception, caused by: > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-14-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:23:52,142 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-2:ctx-219e47df job-99/job-101 ctx-74b313f0) > (logid:0eb71d41) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-14-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM[User|i-2-14-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:23:52,239 DEBUG [c.c.v.UserVmManagerImpl] > (API-Job-Executor-1:ctx-2ddeb2a4 job-99 ctx-f6560edc) (logid:0eb71d41) > Destroying vm VM[User|i-2-14-VM] as it failed to create on Host with Id:null > > 2019-12-01 03:23:52,640 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-1:ctx-2ddeb2a4 job-99) (logid:0eb71d41) Unexpected > exception while executing > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin > > com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due > to insufficient capacity > > Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable > to create a deployment for VM[User|i-2-14-VM]Scope=interface > com.cloud.dc.DataCenter; id=1 > > 2019-12-01 03:23:52,644 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-1:ctx-2ddeb2a4 job-99) (logid:0eb71d41) Complete async > job-99, jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable > to start a VM due to insufficient capacity"} > > 2019-12-01 03:43:27,177 INFO [c.c.a.ApiServer] > (qtp504527234-18:ctx-dce5313f ctx-f343b0ac) (logid:e90a7d5c) Source cidrs > formatting error 0.0.0.0/0.0.0.0 > > 2019-12-01 03:44:41,436 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-6:ctx-446aa701 job-109/job-110 ctx-6755b29a) > (logid:8f5f512c) Cleaning up because we're unabl to implement the network > Ntwk[206|Guest|11] > > com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to > allocate vnet as a part of network Ntwk[206|Guest|11] implement > Scope=interface com.cloud.dc.DataCenter; id=1 > > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > > 2019-12-01 03:44:42,069 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-6:ctx-446aa701 job-109/job-110 ctx-6755b29a) > (logid:8f5f512c) Invocation exception, caused by: > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-15-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:44:42,069 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-6:ctx-446aa701 job-109/job-110 ctx-6755b29a) > (logid:8f5f512c) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-15-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM[User|i-2-15-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:44:43,465 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-7:ctx-e0086abb job-109/job-111 ctx-d2bcce7f) > (logid:8f5f512c) Cleaning up because we're unabl to implement the network > Ntwk[206|Guest|11] > > com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to > allocate vnet as a part of network Ntwk[206|Guest|11] implement > Scope=interface com.cloud.dc.DataCenter; id=1 > > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > > 2019-12-01 03:44:43,948 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-7:ctx-e0086abb job-109/job-111 ctx-d2bcce7f) > (logid:8f5f512c) Invocation exception, caused by: > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-15-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:44:43,948 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-7:ctx-e0086abb job-109/job-111 ctx-d2bcce7f) > (logid:8f5f512c) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-15-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM[User|i-2-15-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 03:44:44,024 DEBUG [c.c.v.UserVmManagerImpl] > (API-Job-Executor-6:ctx-360c0e90 job-109 ctx-3497c21d) (logid:8f5f512c) > Destroying vm VM[User|i-2-15-VM] as it failed to create on Host with Id:null > > 2019-12-01 03:44:44,513 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-6:ctx-360c0e90 job-109) (logid:8f5f512c) Unexpected > exception while executing > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin > > com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due > to insufficient capacity > > Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable > to create a deployment for VM[User|i-2-15-VM]Scope=interface > com.cloud.dc.DataCenter; id=1 > > 2019-12-01 03:44:44,518 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-6:ctx-360c0e90 job-109) (logid:8f5f512c) Complete async > job-109, jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable > to start a VM due to insufficient capacity"} > > 2019-12-01 04:20:29,350 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-14:ctx-c18e434f job-120/job-121 ctx-0e8d39cb) > (logid:04300826) Cleaning up because we're unable to implement the network > Ntwk[206|Guest|11] > > com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to > allocate vnet as a part of network Ntwk[206|Guest|11] implement > Scope=interface com.cloud.dc.DataCenter; id=1 > > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > > 2019-12-01 04:20:29,840 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-14:ctx-c18e434f job-120/job-121 ctx-0e8d39cb) > (logid:04300826) Invocation exception, caused by: > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-17-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 04:20:29,841 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-14:ctx-c18e434f job-120/job-121 ctx-0e8d39cb) > (logid:04300826) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-17-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM[User|i-2-17-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 04:20:31,506 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-15:ctx-a4b1bc3a job-120/job-122 ctx-2b626f79) > (logid:04300826) Cleaning up because we're unable to implement the network > Ntwk[206|Guest|11] > > com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to > allocate vnet as a part of network Ntwk[206|Guest|11] implement > Scope=interface com.cloud.dc.DataCenter; id=1 > > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > > 2019-12-01 04:20:31,984 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-15:ctx-a4b1bc3a job-120/job-122 ctx-2b626f79) > (logid:04300826) Invocation exception, caused by: > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-17-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 04:20:31,984 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-15:ctx-a4b1bc3a job-120/job-122 ctx-2b626f79) > (logid:04300826) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-17-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM[User|i-2-17-VM]Scope=interface com.cloud.dc.DataCenter; > id=1 > > 2019-12-01 04:20:32,111 DEBUG [c.c.v.UserVmManagerImpl] > (API-Job-Executor-9:ctx-e3883c7d job-120 ctx-55f72391) (logid:04300826) > Destroying vm VM[User|i-2-17-VM] as it failed to create on Host with Id:null > > 2019-12-01 04:20:32,494 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-9:ctx-e3883c7d job-120) (logid:04300826) Unexpected > exception while executing > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin > > com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due > to insufficient capacity > > Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable > to create a deployment for VM[User|i-2-17-VM]Scope=interface > com.cloud.dc.DataCenter; id=1 > > 2019-12-01 04:20:32,496 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-9:ctx-e3883c7d job-120) (logid:04300826) Complete async > job-120, jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable > to start a VM due to insufficient capacity"} > > > > Thanks > Pradeep > > >
