Hello Rohit,
I only have one CS management server, and it is 4.19.0.2. The SSVM shows
4.19.0.2. These are XCP-NG hosts. I also just deleted the SSVM to recreate it,
and it is now having the same issues.
I also redownloaded the vhd-util. Same issue. Cloudstack does not show any
error in the console. All is green and happy.
There are no capacity issues. This storage appliance is also used by another
non-CS cluster, and it works fine. I can also go into XOA and create a VM on
the CS cluster without issue using the same SR. And a few more details:
1. CS management server has been reboot, numerous times.
2. DB server has been rebooted.
3. The xen toolstack on all the hosts where restarted.
I also thought about tags. I am not using tags at the moment.
Here is a snippet of the log during the error:
2024-07-09 06:57:20,238 INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Guest VM is requested with
Custom[UEFI] Boot Type false
2024-07-09 06:57:20,238 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Looking for hosts in dc: 3 pod:3
cluster:3
2024-07-09 06:57:20,248 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) List of hosts in descending order of
free capacity in the cluster: [5, 7, 6]
2024-07-09 06:57:20,248 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) FirstFitAllocator has 3 hosts to
check for allocation: [Host
{"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"},
Host
{"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
Host
{"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-0862-4c78-b1e4-d21126f8408c"}]
2024-07-09 06:57:20,264 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Found 3 hosts for allocation after
prioritization: [Host
{"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"},
Host
{"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
Host
{"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-0862-4c78-b1e4-d21126f8408c"}]
2024-07-09 06:57:20,264 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Looking for speed=500Mhz, Ram=1024 MB
2024-07-09 06:57:20,264 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Host name: csc-n1, hostId: 5 is in
avoid set, skipping this and trying other available hosts
2024-07-09 06:57:20,264 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Host name: csc-n3, hostId: 7 is in
avoid set, skipping this and trying other available hosts
2024-07-09 06:57:20,271 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) template 316 is already in store:6, type:Image
2024-07-09 06:57:20,278 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Host: 6 has cpu capability (cpu:88,
speed:2200) to support requested CPU: 1 and requested speed: 500
2024-07-09 06:57:20,278 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Checking if host: 6 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 2.0
2024-07-09 06:57:20,282 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Hosts's actual total CPU: 193600 and
CPU after applying overprovisioning: 387200
2024-07-09 06:57:20,282 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Free CPU: 309300 , Requested CPU: 500
2024-07-09 06:57:20,282 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Free RAM: (641.79 GB) 689115103232 ,
Requested RAM: (1.00 GB) 1073741824
2024-07-09 06:57:20,282 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Host has enough CPU and RAM available
2024-07-09 06:57:20,282 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) STATS: Can alloc CPU from host: 6,
used: 77900, reserved: 0, actual total: 193600, total with overprovisioning:
387200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2024-07-09 06:57:20,282 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) STATS: Can alloc MEM from host: 6,
used: (94.25 GB) 101200166912, reserved: (0 bytes) 0, total: (736.04 GB)
790315270144; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false
, considerReservedCapacity?: true
2024-07-09 06:57:20,282 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Found a suitable host, adding to
list: 6
2024-07-09 06:57:20,282 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369
FirstFitRoutingAllocator) (logid:c56ddd5b) Host Allocator returning 1 suitable
hosts
2024-07-09 06:57:20,282 INFO [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Re-ordering hosts [Host
{"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-0862-4c78-b1e4-d21126f8408c"}]
by priorities {}
2024-07-09 06:57:20,282 INFO [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Hosts after re-ordering are: [Host
{"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-0862-4c78-b1e4-d21126f8408c"}]
2024-07-09 06:57:20,285 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Checking suitable pools for volume (Id, Type): (1000,ROOT)
2024-07-09 06:57:20,289 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Setting pool_id to NULL for volume id=1000 as it is in
Allocated state
2024-07-09 06:57:20,289 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) We need to allocate new storagepool for this volume
2024-07-09 06:57:20,292 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Calling StoragePoolAllocators to find suitable pools
2024-07-09 06:57:20,296 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) System VMs will use shared storage for zone id=3
2024-07-09 06:57:20,296 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) LocalStoragePoolAllocator is returning null since the disk
profile does not use local storage and bypassStorageTypeCheck is false.
2024-07-09 06:57:20,296 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Looking for pools in dc [3], pod [3] and cluster [3]. Disabled
pools will be ignored.
2024-07-09 06:57:20,301 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Found pools [[Pool[14|NetworkFilesystem],
Pool[18|NetworkFilesystem]]] that match with tags [[]].
2024-07-09 06:57:20,303 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Checking if storage pool
[{"name":"SCALE-CL1-Primary","uuid":"de8ad9b7-93a7-31d0-9646-075b686b0d04"}] is
suitable to disk [DskChr[ROOT|5242880000|]].
2024-07-09 06:57:20,303 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) StoragePool
[{"name":"SCALE-CL1-Primary","uuid":"de8ad9b7-93a7-31d0-9646-075b686b0d04"}] is
in avoid set, skipping this pool to allocation of disk
[DskChr[ROOT|5242880000|]].
2024-07-09 06:57:20,303 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Adding storage pool [Pool[14|NetworkFilesystem]] to avoid set
during allocation of disk [DskChr[ROOT|5242880000|]].
2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Checking if storage pool
[{"name":"TN-2-Prod","uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff"}] is
suitable to disk [DskChr[ROOT|5242880000|]].
2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) StoragePool
[{"name":"TN-2-Prod","uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff"}] is in
avoid set, skipping this pool to allocation of disk [DskChr[ROOT|5242880000|]].
2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Adding storage pool [Pool[18|NetworkFilesystem]] to avoid set
during allocation of disk [DskChr[ROOT|5242880000|]].
2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) ClusterScopeStoragePoolAllocator is returning [0] suitable
storage pools [[]].
2024-07-09 06:57:20,306 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Could not find any zone wide storage pool that matched with
any of the following tags [[]].
2024-07-09 06:57:20,308 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) ZoneWideStoragePoolAllocator is returning [0] suitable storage
pools [[]].
2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) No suitable pools found for volume:
Vol[1000|name=ROOT-598|vm=598|ROOT] under cluster: 3
2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) No suitable pools found
2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) No suitable storagePools found under this Cluster: 3
2024-07-09 06:57:20,308 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) copyAsync inspecting src type TEMPLATE copyAsync inspecting
dest type VOLUME
2024-07-09 06:57:20,312 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Could not find suitable Deployment Destination for this VM
under any clusters, returning.
2024-07-09 06:57:20,313 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Searching all possible resources under this Zone: 3
2024-07-09 06:57:20,315 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Listing clusters in order of aggregate capacity, that have (at
least one host with) enough CPU and RAM capacity under this Zone: 3
2024-07-09 06:57:20,317 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Removing from the clusterId list these clusters from avoid
set: [3]
2024-07-09 06:57:20,318 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) No clusters found after removing disabled clusters and
clusters in avoid list, returning.
2024-07-09 06:57:20,337 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) VM instance
{"id":598,"instanceName":"v-598-VM","type":"ConsoleProxy","uuid":"5479ccfb-32ec-405e-9e6b-b29406b954e3"}
state transited from [Starting] to [Stopped] with event [OperationFailed].
VM's original host: null, new host: null, host before state transition: Host
{"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"}
2024-07-09 06:57:20,345 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CopyCommand
2024-07-09 06:57:20,349 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Hosts's actual total CPU: 193600 and CPU after applying
overprovisioning: 387200
2024-07-09 06:57:20,349 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Hosts's actual total RAM: (736.04 GB) 790315265792 and RAM
after applying overprovisioning: (736.04 GB) 790315270144
2024-07-09 06:57:20,349 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) release cpu from host: 7, old used: 55500,reserved: 0, actual
total: 193600, total with overprovisioning: 387200; new used: 55000,reserved:0;
movedfromreserved: false,moveToReserveredfalse
2024-07-09 06:57:20,349 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) release mem from host: 7, old used: (55.25 GB)
59324235776,reserved: (0 bytes) 0, total: (736.04 GB) 790315270144; new used:
(54.25 GB) 58250493952,reserved:(0 bytes) 0; movedfromreserved:
false,moveToReserveredfalse
2024-07-09 06:57:20,350 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) We are returning the default host to execute commands because
the source and destination objects are not NFS type.
2024-07-09 06:57:20,354 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Seq 6-4921308492809109519: Sending { Cmd , MgmtId:
345052750974, via: 6(csc-n2), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"13f86c22-a422-459b-90d2-450a145724b3","origUrl":http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.0-xen.vhd.bz2,"uuid":"a6bebefa-67f3-4946-b5e2-f1053f37810b","id":"316","format":"VHD","accountId":"2","checksum":"{SHA-512}e692cfa4925bf5e5de6f0eee50462b89bbbd84360d5d14da85e417a938fef476032c2b889b72225fe232be7212fbde67db1135f088fddff2c5f5567958285b6e","hvm":"true","displayText":"ystemvm-xenserver-4.19.0","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff","name":"TN-2-Prod","id":"18","poolType":"NetworkFilesystem","host":"172.16.27.27","path":"/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary","port":"2049","url":"NetworkFilesystem://172.16.27.27/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary/?ROLE=Primary&STOREUUID=e68d4f6d-8539-3c88-a9a2-af63726bdeff","isManaged":"false"}},"name":"316-2-9bd66109-0e9a-3e83-a798-a6afe03384b2","size":";(4.88
GB)
5242880000","hypervisorType":"XenServer","bootable":"false","uniqueName":"316-2-9bd66109-0e9a-3e83-a798-a6afe03384b2","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c3dac8c-bbf5-4d5c-a5bd-3d2620e4f14c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff","name":"TN-2-Prod","id":"18","poolType":"NetworkFilesystem","host":"172.16.27.27","path":"/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary","port":"2049","url":"NetworkFilesystem://172.16.27.27/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary/?ROLE=Primary&STOREUUID=e68d4f6d-8539-3c88-a9a2-af63726bdeff","isManaged":"false"}},"name":"ROOT-599","size":"(4.88
GB)
5242880000","volumeId":"1003","vmName":"s-599-VM","accountId":"1","format":"VHD","provisioningType":"THIN","poolId":"18","id":"1003","deviceId":"0","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}]
}
2024-07-09 06:57:20,355 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":598,"instanceName":"v-598-VM","type":"ConsoleProxy","uuid":"5479ccfb-32ec-405e-9e6b-b29406b954e3"}Scope=interface
com.cloud.dc.DataCenter; id=3
2024-07-09 06:57:20,355 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
(logid:c56ddd5b) Rethrow exception
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":598,"instanceName":"v-598-VM","type":"ConsoleProxy","uuid":"5479ccfb-32ec-405e-9e6b-b29406b954e3"}Scope=interface
com.cloud.dc.DataCenter; id=3
2024-07-09 06:57:20,355 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Done with
run of VM work job: com.cloud.vm.VmWorkStart for VM 598, job origin: 7483
2024-07-09 06:57:20,355 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Unable to
complete AsyncJobVO: {id:7492, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAACVnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 345052750974, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Tue Jul 09 06:57:17 UTC 2024, removed: null}, job
origin:7483
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":598,"instanceName":"v-598-VM","type":"ConsoleProxy","uuid":"5479ccfb-32ec-405e-9e6b-b29406b954e3"}Scope=interface
com.cloud.dc.DataCenter; id=3
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1228)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5457)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5581)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-09 06:57:20,356 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Seq 6-4921308492809109519: Executing: { Cmd , MgmtId:
345052750974, via: 6(csc-n2), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"13f86c22-a422-459b-90d2-450a145724b3","origUrl":http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.0-xen.vhd.bz2,"uuid":"a6bebefa-67f3-4946-b5e2-f1053f37810b","id":"316","format":"VHD","accountId":"2","checksum":"{SHA-512}e692cfa4925bf5e5de6f0eee50462b89bbbd84360d5d14da85e417a938fef476032c2b889b72225fe232be7212fbde67db1135f088fddff2c5f5567958285b6e","hvm":"true","displayText":"ystemvm-xenserver-4.19.0","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff","name":"TN-2-Prod","id":"18","poolType":"NetworkFilesystem","host":"172.16.27.27","path":"/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary","port":"2049","url":"NetworkFilesystem://172.16.27.27/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary/?ROLE=Primary&STOREUUID=e68d4f6d-8539-3c88-a9a2-af63726bdeff","isManaged":"false"}},"name":"316-2-9bd66109-0e9a-3e83-a798-a6afe03384b2","size":";(4.88
GB)
5242880000","hypervisorType":"XenServer","bootable":"false","uniqueName":"316-2-9bd66109-0e9a-3e83-a798-a6afe03384b2","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c3dac8c-bbf5-4d5c-a5bd-3d2620e4f14c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff","name":"TN-2-Prod","id":"18","poolType":"NetworkFilesystem","host":"172.16.27.27","path":"/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary","port":"2049","url":"NetworkFilesystem://172.16.27.27/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary/?ROLE=Primary&STOREUUID=e68d4f6d-8539-3c88-a9a2-af63726bdeff","isManaged":"false"}},"name":"ROOT-599","size":"(4.88
GB)
5242880000","volumeId":"1003","vmName":"s-599-VM","accountId":"1","format":"VHD","provisioningType":"THIN","poolId":"18","id":"1003","deviceId":"0","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}]
}
2024-07-09 06:57:20,356 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-29:ctx-d7829706) (logid:faadd309) Seq 6-4921308492809109519:
Executing request
2024-07-09 06:57:20,357 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase]
(DirectAgent-29:ctx-d7829706) (logid:efafa159) Executing command CopyCommand:
[{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"13f86c22-a422-459b-90d2-450a145724b3","origUrl":http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.0-xen.vhd.bz2,"uuid":"a6bebefa-67f3-4946-b5e2-f1053f37810b","id":316,"format":"VHD","accountId":2,"checksum":"{SHA-512}e692cfa4925bf5e5de6f0eee50462b89bbbd84360d5d14da85e417a938fef476032c2b889b72225fe232be7212fbde67db1135f088fddff2c5f5567958285b6e","hvm":true,"displayText":"ystemvm-xenserver-4.19.0","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff","name":"TN-2-Prod","id":18,"poolType":"NetworkFilesystem","host":"172.16.27.27","path":"/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary","port":2049,"url":"NetworkFilesystem://172.16.27.27/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary/?ROLE=Primary&STOREUUID=e68d4f6d-8539-3c88-a9a2-af63726bdeff","isManaged":false}},"name":"316-2-9bd66109-0e9a-3e83-a798-a6afe03384b2","size":5242880000,"hypervisorType":"XenServer","bootable":false,"uniqueName":"316-2-9bd66109-0e9a-3e83-a798-a6afe03384b2","directDownload":false,"deployAsIs":false,"followRedirects":false}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c3dac8c-bbf5-4d5c-a5bd-3d2620e4f14c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e68d4f6d-8539-3c88-a9a2-af63726bdeff","name":"TN-2-Prod","id":18,"poolType":"NetworkFilesystem","host":"172.16.27.27","path":"/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary","port":2049,"url":"NetworkFilesystem://172.16.27.27/mnt/tn2-prod/Rudio_VPC/flx-lv-cl1-primary/?ROLE=Primary&STOREUUID=e68d4f6d-8539-3c88-a9a2-af63726bdeff","isManaged":false}},"name":"ROOT-599","size":5242880000,"volumeId":1003,"vmName":"s-599-VM","accountId":1,"format":"VHD","provisioningType":"THIN","poolId":18,"id":1003,"deviceId":0,"hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":false}},"executeInSequence":true,"options":{},"options2":{},"wait":0,"bypassHostMaintenance":false}].
2024-07-09 06:57:20,361 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Complete
async job-7492, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyADdjb20uY2xvdWQuZXhjZXB0aW9uLkluc3VmZmljaWVudFNlcnZlckNhcGFjaXR5RXhjZXB0aW9uVk1PcAAAABACAAFaABVhZmZpbml0eUdyb3Vwc0FwcGxpZWR4cgAxY29tLmNsb3VkLmV4Y2VwdGlvbi5JbnN1ZmZpY2llbnRDYXBhY2l0eUV4Y2VwdGlvblZNT3AAAAAVAgACTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc2NvcGV0ABFMamF2YS9sYW5nL0NsYXNzO3hyACJjb20uY2xvdWQuZXhjZXB0aW9uLkNsb3VkRXhjZXB0aW9ueeiOG_HkRMcCAAJMAAtjc0Vycm9yQ29kZXQAE0xqYXZhL2xhbmcvSW50ZWdlcjtMAAZpZExpc3R0ABVMamF2YS91dGlsL0FycmF5TGlzdDt4cgATamF2YS5sYW5nLkV4Y2VwdGlvbtD9Hz4aOxzEAgAAeHIAE2phdmEubGFuZy5UaHJvd2FibGXVxjUnOXe4ywMABEwABWNhdXNldAAVTGphdmEvbGFuZy9UaHJvd2FibGU7TAANZGV0YWlsTWVzc2FnZXQAEkxqYXZhL2xhbmcvU3RyaW5nO1sACnN0YWNrVHJhY2V0AB5bTGphdmEvbGFuZy9TdGFja1RyYWNlRWxlbWVudDtMABRzdXBwcmVzc2VkRXhjZXB0aW9uc3QAEExqYXZhL3V0aWwvTGlzdDt4cHEAfgANdACWVW5hYmxlIHRvIGNyZWF0ZSBhIGRlcGxveW1lbnQgZm9yIFZNIGluc3RhbmNlIHsiaWQiOjU5OCwiaW5zdGFuY2VOYW1lIjoidi01OTgtVk0iLCJ0eXBlIjoiQ29uc29sZVByb3h5IiwidXVpZCI6IjU0NzljY2ZiLTMyZWMtNDA1ZS05ZTZiLWIyOTQwNmI5NTRlMyJ9dXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAVc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAIQgAGZm9ybWF0SQAKbGluZU51bWJlckwAD2NsYXNzTG9hZGVyTmFtZXEAfgAKTAAOZGVjbGFyaW5nQ2xhc3NxAH4ACkwACGZpbGVOYW1lcQB-AApMAAptZXRob2ROYW1lcQB-AApMAAptb2R1bGVOYW1lcQB-AApMAA1tb2R1bGVWZXJzaW9ucQB-AAp4cAEAAATMdAADYXBwdAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0ABBvcmNoZXN0cmF0ZVN0YXJ0cHBzcQB-ABEBAAAVUXEAfgATcQB-ABRxAH4AFXEAfgAWcHBzcQB-ABEC_____nB0AC1qZGsuaW50ZXJuYWwucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YXQAB2ludm9rZTB0AAlqYXZhLmJhc2V0AAcxMS4wLjIzc3EAfgARAgAAAD5wcQB-ABlxAH4AGnQABmludm9rZXEAfgAccQB-AB1zcQB-ABECAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-AB9xAH4AHHEAfgAdc3EAfgARAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AH3EAfgAccQB-AB1zcQB-ABEBAAAAa3EAfgATdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-ABEBAAAVzXEAfgATcQB-ABRxAH4AFXEAfgApcHBzcQB-ABEBAAAAZnEAfgATdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4AEQEAAAKOcQB-ABN0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4AEQEAAAAwcQB-ABN0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgARAQAAADdxAH4AE3QAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgARAQAAAGZxAH4AE3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AOXQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgARAQAAADRxAH4AE3EAfgA8cQB-ADl0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgARAQAAAC1xAH4AE3QAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgA1cQB-ADZwcHNxAH4AEQEAAAJacQB-ABNxAH4AMHEAfgAxcQB-ADZwcHNxAH4AEQIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgA6cQB-ABxxAH4AHXNxAH4AEQIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ANnEAfgAccQB-AB1zcQB-ABECAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAccQB-AB1zcQB-ABECAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEtxAH4ANnEAfgAccQB-AB1zcQB-ABECAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ANnEAfgAccQB-AB1zcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4c3IAEWphdmEubGFuZy5JbnRlZ2VyEuKgpPeBhzgCAAFJAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAEO9zcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHEAfgBVAAAAAAAAAAN2cgAXY29tLmNsb3VkLmRjLkRhdGFDZW50ZXIQUFRtA0UeAAIAAHhwAA
2024-07-09 06:57:20,361 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-29:ctx-d7829706) (logid:efafa159) Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
13f86c22-a422-459b-90d2-450a145724b3 failed due to The uuid you supplied was
invalid.
2024-07-09 06:57:20,361 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-29:ctx-d7829706) (logid:efafa159) Unable to create volume;
Pool=volumeTO[uuid=5c3dac8c-bbf5-4d5c-a5bd-3d2620e4f14c|path=null|datastore=PrimaryDataStoreTO[uuid=e68d4f6d-8539-3c88-a9a2-af63726bdeff|name=TN-2-Prod|id=18|pooltype=NetworkFilesystem]];
Disk:
com.cloud.utils.exception.CloudRuntimeException: Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
13f86c22-a422-459b-90d2-450a145724b3 failed due to The uuid you supplied was
invalid.
at
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getVDIbyUuid(XenServerStorageProcessor.java:655)
at
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.cloneVolumeFromBaseTemplate(XenServerStorageProcessor.java:843)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:106)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:64)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1789)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: The uuid you supplied was invalid.
at com.xensource.xenapi.Types.checkResponse(Types.java:1491)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:458)
... 21 more
2024-07-09 06:57:20,361 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-29:ctx-d7829706) (logid:efafa159) Seq 6-4921308492809109519:
Response Received:
2024-07-09 06:57:20,362 DEBUG [c.c.a.t.Request] (DirectAgent-29:ctx-d7829706)
(logid:efafa159) Seq 6-4921308492809109519: Processing: { Ans: , MgmtId:
345052750974, via: 6(csc-n2), Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for
uuid: 13f86c22-a422-459b-90d2-450a145724b3 failed due to The uuid you supplied
was invalid.","wait":"0","bypassHostMaintenance":"false"}}] }
2024-07-09 06:57:20,362 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-29:ctx-d7829706) (logid:efafa159) Seq 6-4921308492809109519: No
more commands found
2024-07-09 06:57:20,362 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Seq 6-4921308492809109519: Received: { Ans: , MgmtId:
345052750974, via: 6(csc-n2), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2024-07-09 06:57:20,363 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Publish
async job-7492 complete on message bus
2024-07-09 06:57:20,363 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Wake up
jobs related to job-7492
2024-07-09 06:57:20,363 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Update db
status for job-7492
2024-07-09 06:57:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Wake up
jobs joined with job-7492 and disjoin all subjobs created from job- 7492
2024-07-09 06:57:20,370 ERROR [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Unable to create volume
[{"name":"ROOT-599","uuid":"5c3dac8c-bbf5-4d5c-a5bd-3d2620e4f14c"}] due to
[com.cloud.utils.exception.CloudRuntimeException: Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
13f86c22-a422-459b-90d2-450a145724b3 failed due to The uuid you supplied was
invalid.].
2024-07-09 06:57:20,371 WARN [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:18] is
unreachable: Unable to create volume
[{"name":"ROOT-599","uuid":"5c3dac8c-bbf5-4d5c-a5bd-3d2620e4f14c"}] due to
[com.cloud.utils.exception.CloudRuntimeException: Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
13f86c22-a422-459b-90d2-450a145724b3 failed due to The uuid you supplied was
invalid.].
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1814)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1923)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.prepare(Unknown Source)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1268)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5457)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5581)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-09 06:57:20,374 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Cleaning up resources for the vm VM instance
{"id":599,"instanceName":"s-599-VM","type":"SecondaryStorageVm","uuid":"cdbb5227-78fb-4cd4-aadd-0ff6d2639dce"}
in Starting state
2024-07-09 06:57:20,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Done
executing com.cloud.vm.VmWorkStart for job-7492
2024-07-09 06:57:20,383 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492) (logid:c56ddd5b) Remove
job-7492 from job monitoring
2024-07-09 06:57:20,391 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Seq 7-4779726579523649556: Sending { Cmd , MgmtId:
345052750974, via: 7(csc-n3), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.184.146","forceStop":"false","volumesToDisconnect":[],"vmName":"s-599-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-07-09 06:57:20,391 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-5:ctx-b19182a5 job-7477/job-7495 ctx-8d8d7980)
(logid:efafa159) Seq 7-4779726579523649556: Executing: { Cmd , MgmtId:
345052750974, via: 7(csc-n3), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.184.146","forceStop":"false","volumesToDisconnect":[],"vmName":"s-599-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-07-09 06:57:20,391 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-16:ctx-95127cbf) (logid:78c55027) Seq 7-4779726579523649556:
Executing request
2024-07-09 06:57:20,392 WARN [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-e0048687) (logid:6567e8d2) Unable to start proxy [598] due
to [Unable to create a deployment for VM instance
{"id":598,"instanceName":"v-598-VM","type":"ConsoleProxy","uuid":"5479ccfb-32ec-405e-9e6b-b29406b954e3"}].
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":598,"instanceName":"v-598-VM","type":"ConsoleProxy","uuid":"5479ccfb-32ec-405e-9e6b-b29406b954e3"}Scope=interface
com.cloud.dc.DataCenter; id=3
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1228)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5457)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5581)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-09 06:57:20,392 INFO [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-e0048687) (logid:6567e8d2) Unable to start console proxy vm
for standby capacity, vm id : 598, will recycle it and start a new one
From: Rohit Yadav <[email protected]>
Date: Tuesday, July 9, 2024 at 12:44?AM
To: [email protected] <[email protected]>
Subject: Re: Unable to create new VMs or Console Proxy
WARNING: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
Hi Marty,
Can you check and ensure all your CloudStack hosts (management server, kvm
hosts if applicable and ssvm, cpvm) are on the same version (4.19.0.2) &
restarted post update/upgrade.
Once you've done this, can you share your management server logs where you see
the exceptions/errors around VM deployment. You may also want to check if
you've any capacity issues, or using any offering that's tagged (host or
storage tags) for which you don't have available capacity?
Regards.
________________________________
From: Marty Godsey <[email protected]>
Sent: Tuesday, July 9, 2024 07:29
To: [email protected] <[email protected]>
Subject: Re: Unable to create new VMs or Console Proxy
Ok, figured, out the service offerings issues, I rebooted the database and it
came back. I still can not create VMs.
It is stating that the storage is suitable, but it is in avoid set. I have no
idea why it is in avoid set since all hosts and the Cloudstack management
server can get to it.
Where would I check this?
From: Marty Godsey <[email protected]>
Date: Monday, July 8, 2024 at 9:19?PM
To: [email protected] <[email protected]>
Subject: Unable to create new VMs or Console Proxy
WARNING: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
Question. All of a sudden, I cannot create new VMs, and the console proxy also
can't be created. As I spent the past hour looking, I have found that in the
console, when I go to offerings for compute, disk, or system, I get a 404, and
it says it can't find anything for domain -1. But when I look in the database,
I see the offerings. I took this opportunity to update to 4.19.0.2 since I was
already on 4.19.0.1. At first I thought it was storage but it is not. Running
VMS and secStorage and VRs are all running fine.
Where should I look from here? Nothing was changed for the root domain.
