Hi guys,
We did an upgrade from 4.15 to 4.18 and everything went smoothly! We can create
new vms without any issue however if I stop previously created vms they won’t
start again. Any Clue?
*
Hyp KVM
*
ACS 4.18.0
*
Primary Storage Ceph
Logs
2023-05-19 19:17:00,878 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Deploy avoids pods: null, clusters: null, hosts: null
2023-05-19 19:17:00,882 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) VM start attempt #1
2023-05-19 19:17:00,887 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Root volume is ready, need to place VM in volume's cluster
2023-05-19 19:17:00,887 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Vol[1607|name=ROOT-686|vm=686|ROOT] is READY, changing
deployment plan to use this pool's dcId: 2 , podId: null , and clusterId: null
2023-05-19 19:17:00,909 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Adding pods to avoid lists for non-explicit VM deployment: []
2023-05-19 19:17:00,909 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Adding clusters to avoid lists for non-explicit VM deployment:
[]
2023-05-19 19:17:00,909 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Adding hosts to avoid lists for non-explicit VM deployment: []
2023-05-19 19:17:00,909 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) DeploymentPlanner allocation algorithm: null
2023-05-19 19:17:00,909 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Trying to allocate a host and storage pools from dc:2,
pod:null,cluster:null, requested cpu: 4800, requested ram: (8.00 GB) 8589934592
2023-05-19 19:17:00,909 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Is ROOT volume READY (pool already allocated)?: Yes
2023-05-19 19:17:00,924 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Deploy avoids pods: [], clusters: [], hosts: []
2023-05-19 19:17:00,924 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Deploy hosts with priorities {} , hosts have NORMAL priority
by default
2023-05-19 19:17:00,926 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) This VM has last host_id specified, trying to choose the same
host: 2
2023-05-19 19:17:00,933 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Host {id: 2, name:co-centro1a-phy-k1-hyp-01.kuasar.cloud,
uuid: ba87d8ff-21e0-4f20-8bda-b8fa76453a1c} is KVM hypervisor type, no max
guest limit check needed
2023-05-19 19:17:00,950 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Host: 2 doesn't have cpu capability (cpu:128, speed:2399) to
support requested CPU: 2 and requested speed: 2400
2023-05-19 19:17:00,950 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) The last host of this VM does not have enough capacity
2023-05-19 19:17:00,950 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Cannot choose the last host to deploy this VM
2023-05-19 19:17:00,955 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Searching all possible resources under this Zone: 2
2023-05-19 19:17:00,963 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Listing clusters in order of aggregate capacity, that have (at
least one host with) enough CPU and RAM capacity under this Zone: 2
2023-05-19 19:17:00,976 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Removing from the clusterId list these clusters from avoid
set: []
2023-05-19 19:17:01,010 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) The clusterId list for the given offering tag: [1]
2023-05-19 19:17:01,017 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Checking resources in Cluster: 1 under Pod: 3
2023-05-19 19:17:01,022 INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Guest VM is requested with
Custom[UEFI] Boot Type false
2023-05-19 19:17:01,023 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Looking for hosts in dc: 2 pod:3
cluster:1
2023-05-19 19:17:01,023 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Looking for hosts having tag
specified on SvcOffering:k1
2023-05-19 19:17:01,033 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Hosts with tag 'k1' are:[Host
{"id":2,"name":"co-centro1a-phy-k1-hyp-01.kuasar.cloud","type":"Routing","uuid":"ba87d8ff-21e0-4f20-8bda-b8fa76453a1c"}]
2023-05-19 19:17:01,037 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) FirstFitAllocator has 1 hosts to
check for allocation: [Host
{"id":2,"name":"co-centro1a-phy-k1-hyp-01.kuasar.cloud","type":"Routing","uuid":"ba87d8ff-21e0-4f20-8bda-b8fa76453a1c"}]
2023-05-19 19:17:01,047 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Found 1 hosts for allocation after
prioritization: [Host
{"id":2,"name":"co-centro1a-phy-k1-hyp-01.kuasar.cloud","type":"Routing","uuid":"ba87d8ff-21e0-4f20-8bda-b8fa76453a1c"}]
2023-05-19 19:17:01,047 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Looking for speed=4800Mhz, Ram=8192
MB
2023-05-19 19:17:01,047 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Host {id: 2,
name:co-centro1a-phy-k1-hyp-01.kuasar.cloud, uuid:
ba87d8ff-21e0-4f20-8bda-b8fa76453a1c} is KVM hypervisor type, no max guest
limit check needed
2023-05-19 19:17:01,059 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Host: 2 doesn't have cpu capability
(cpu:128, speed:2399) to support requested CPU: 2 and requested speed: 2400
2023-05-19 19:17:01,059 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Checking if host: 2 has enough
capacity for requested CPU: 4800 and requested RAM: (8.00 GB) 8589934592 ,
cpuOverprovisioningFactor: 6.0
2023-05-19 19:17:01,063 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Hosts's actual total CPU: 307072 and
CPU after applying overprovisioning: 1842432
2023-05-19 19:17:01,063 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Free CPU: 1623976 , Requested CPU:
4800
2023-05-19 19:17:01,063 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Free RAM: (726.06 GB) 779597905920 ,
Requested RAM: (8.00 GB) 8589934592
2023-05-19 19:17:01,063 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Host has enough CPU and RAM available
2023-05-19 19:17:01,063 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) STATS: Can alloc CPU from host: 2,
used: 199256, reserved: 19200, actual total: 307072, total with
overprovisioning: 1842432; requested cpu:4800,alloc_from_last_host?:false
,considerReservedCapacity?: true
2023-05-19 19:17:01,063 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) STATS: Can alloc MEM from host: 2,
used: (248.74 GB) 267084890112, reserved: (32.00 GB) 34359738368, total:
(1006.80 GB) 1081042534400; requested mem: (8.00 GB) 8589934592,
alloc_from_last_host?: false , considerReservedCapacity?: true
2023-05-19 19:17:01,063 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Not using host 2; host has cpu
capability? false, host has capacity?true
2023-05-19 19:17:01,064 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611
FirstFitRoutingAllocator) (logid:6edb3ce9) Host Allocator returning 0 suitable
hosts
2023-05-19 19:17:01,064 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) No suitable hosts found
2023-05-19 19:17:01,064 INFO [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Re-ordering hosts [] by priorities {}
2023-05-19 19:17:01,064 INFO [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Hosts after re-ordering are: []
2023-05-19 19:17:01,064 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) No suitable hosts found under this Cluster: 1
2023-05-19 19:17:01,066 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Could not find suitable Deployment Destination for this VM
under any clusters, returning.
2023-05-19 19:17:01,070 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Searching all possible resources under this Zone: 2
2023-05-19 19:17:01,073 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Listing clusters in order of aggregate capacity, that have (at
least one host with) enough CPU and RAM capacity under this Zone: 2
2023-05-19 19:17:01,079 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Removing from the clusterId list these clusters from avoid
set: [1]
2023-05-19 19:17:01,083 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) The clusterId list for the given offering tag: []
2023-05-19 19:17:01,083 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) No clusters found after removing disabled clusters and
clusters in avoid list, returning.
2023-05-19 19:17:01,085 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) VM start attempt #2
2023-05-19 19:17:01,103 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Adding pods to avoid lists for non-explicit VM deployment: []
2023-05-19 19:17:01,104 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Adding clusters to avoid lists for non-explicit VM deployment:
[]
2023-05-19 19:17:01,104 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Adding hosts to avoid lists for non-explicit VM deployment: []
2023-05-19 19:17:01,104 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) DeploymentPlanner allocation algorithm: null
2023-05-19 19:17:01,104 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Trying to allocate a host and storage pools from dc:2,
pod:3,cluster:null, requested cpu: 4800, requested ram: (8.00 GB) 8589934592
2023-05-19 19:17:01,104 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Is ROOT volume READY (pool already allocated)?: No
2023-05-19 19:17:01,117 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Deploy avoids pods: [], clusters: [1], hosts: [2]
2023-05-19 19:17:01,117 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Deploy hosts with priorities {} , hosts have NORMAL priority
by default
2023-05-19 19:17:01,119 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) This VM has last host_id specified, trying to choose the same
host: 2
2023-05-19 19:17:01,126 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) The last host of this VM is in avoid set
2023-05-19 19:17:01,126 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Cannot choose the last host to deploy this VM
2023-05-19 19:17:01,128 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Searching resources only under specified Pod: 3
2023-05-19 19:17:01,133 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Listing clusters in order of aggregate capacity, that have (at
least one host with) enough CPU and RAM capacity under this Pod: 3
2023-05-19 19:17:01,141 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Removing from the clusterId list these clusters from avoid
set: [1]
2023-05-19 19:17:01,145 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) The clusterId list for the given offering tag: []
2023-05-19 19:17:01,145 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) No clusters found after removing disabled clusters and
clusters in avoid list, returning.
2023-05-19 19:17:01,179 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) VM instance
{"id":686,"instanceName":"i-31-686-VM","type":"User","uuid":"d7a8a80a-bef4-4778-9f3c-fc9c06b2cfe1"}
state transited from [Starting] to [Stopped] with event [OperationFailed].
VM's original host: Host
{"id":2,"name":"co-centro1a-phy-k1-hyp-01.kuasar.cloud","type":"Routing","uuid":"ba87d8ff-21e0-4f20-8bda-b8fa76453a1c"},
new host: null, host before state transition: null
2023-05-19 19:17:01,198 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Invocation exception, caused
by:com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":686,"instanceName":"i-31-686-VM","type":"User","uuid":"d7a8a80a-bef4-4778-9f3c-fc9c06b2cfe1"}Scope=interfacecom.cloud.dc.DataCenter;
id=2
2023-05-19 19:17:01,198 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097 ctx-593b1611)
(logid:6edb3ce9) Rethrow
exceptioncom.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM instance
{"id":686,"instanceName":"i-31-686-VM","type":"User","uuid":"d7a8a80a-bef4-4778-9f3c-fc9c06b2cfe1"}Scope=interfacecom.cloud.dc.DataCenter;
id=2
2023-05-19 19:17:01,198 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-28:ctx-f26a211c job-18095/job-18097) (logid:6edb3ce9) Done
with run of VM work job:com.cloud.vm.VmWorkStart for VM 686, job origin: 18095
BR,
Ricardo Pertuz
