Hi Marty, My guess here is that the template `13f86c22-a422-459b-90d2-450a145724b3` from which you are creating a VM is missing on your primary storage, but CloudStack has the information that it is there. To confirm this you can check: - if `13f86c22-a422-459b-90d2-450a145724b3` exist on your primary storage - from the CloudStack DB check if the template was downloaded on your primary storage - `select * from template_spool_ref where local_path="13f86c22-a422-459b-90d2-450a145724b3";` Best regards, Slavka
On Tue, Jul 9, 2024 at 10:39 AM Wei ZHOU <[email protected]> wrote: > Hi Marty, > > It looks like the issue is not caused by server capacity, but storage. > > 2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) > (logid:c56ddd5b) ClusterScopeStoragePoolAllocator is returning [0] > suitable storage pools [[]]. > 2024-07-09 06:57:20,306 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) > (logid:c56ddd5b) Could not find any zone wide storage pool that > matched with any of the following tags [[]]. > 2024-07-09 06:57:20,308 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) > (logid:c56ddd5b) ZoneWideStoragePoolAllocator is returning [0] > suitable storage pools [[]]. > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) > (logid:c56ddd5b) No suitable pools found for volume: > Vol[1000|name=ROOT-598|vm=598|ROOT] under cluster: 3 > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) > (logid:c56ddd5b) No suitable pools found > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) > (logid:c56ddd5b) No suitable storagePools found under this Cluster: 3 > > > > On Tue, Jul 9, 2024 at 9:27 AM Marty Godsey <[email protected]> wrote: > > > > I am also getting this which is strange: Whey does it say there is not > enough capacity. > > > > 2024-07-09 01:08:22,201 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) VM instance > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy","uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"} > state transited from [Starting] to [Starting] with event [OperationRetry]. > VM's original host: null, new host: Host > {"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, > host before state transition: null > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying > overprovisioning: 387200 > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) We are allocating VM, increasing the used capacity of this > host:5 > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Current Used CPU: 41400 , Free CPU:345800 ,Requested CPU: > 500 > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Current Used RAM: (49.00 GB) 52613349376 , Free > RAM:(687.04 GB) 737701920768 ,Requested RAM: (1.00 GB) 1073741824 > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) CPU STATS after allocation: for host: 5, old used: 41400, > old reserved: 0, actual total: 193600, total with overprovisioning: 387200; > new used:41900, reserved:0; requested cpu:500,alloc_from_last:false > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) RAM STATS after allocation: for host: 5, old used: (49.00 > GB) 52613349376, old reserved: (0 bytes) 0, total: (736.04 GB) > 790315270144; new used: (50.00 GB) 53687091200, reserved: (0 bytes) 0; > requested mem: (1.00 GB) 1073741824,alloc_from_last:false > > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Host: 5 has cpu capability (cpu:88, speed:2200) to support > requested CPU: 1 and requested speed: 500 > > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Checking if host: 5 has enough capacity for requested CPU: > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 2.0 > > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying > overprovisioning: 387200 > > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) We need to allocate to the last host again, so checking if > there is enough reserved capacity > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Reserved CPU: 0 , Requested CPU: 500 > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Reserved RAM: (0 bytes) 0 , Requested RAM: (1.00 GB) > 1073741824 > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) STATS: Failed to alloc resource from host: 5 reservedCpu: > 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (1.00 GB) > 1073741824 > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Host does not have enough reserved CPU available, cannot > allocate to this host. > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Checking if host: 5 has enough capacity for requested CPU: > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 2.0 > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying > overprovisioning: 387200 > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Free CPU: 345800 , Requested CPU: 500 > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Free RAM: (687.04 GB) 737701920768 , Requested RAM: (1.00 > GB) 1073741824 > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Host has enough CPU and RAM available > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) STATS: Can alloc CPU from host: 5, used: 41400, reserved: > 0, actual total: 193600, total with overprovisioning: 387200; requested > cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) STATS: Can alloc MEM from host: 5, used: (49.00 GB) > 52613349376, reserved: (0 bytes) 0, total: (736.04 GB) 790315270144; > requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , > considerReservedCapacity?: true > > 2024-07-09 01:08:22,234 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Network id=208 is already implemented > > 2024-07-09 01:08:22,281 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Network id=210 is already implemented > > 2024-07-09 01:08:22,451 DEBUG [o.a.c.h.HAManagerImpl] > (BackgroundTaskPollManager-4:ctx-3dafbb92) (logid:72e1d259) HA health check > task is running... > > 2024-07-09 01:08:22,527 DEBUG [c.c.n.g.ControlNetworkGuru] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Reserved NIC for v-466-VM [ipv4:169.254.17.78 > netmask:255.255.0.0 gateway:169.254.0.1] > > 2024-07-09 01:08:22,544 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Network id=209 is already implemented > > 2024-07-09 01:08:22,564 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Releasing ip address for instance=1462 > > 2024-07-09 01:08:22,574 DEBUG [c.c.n.g.PodBasedNetworkGuru] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Allocated a nic NicProfile > {"broadcastUri":null,"iPv4Address":"10.61.0.91","id":1462,"reservationId":"cc4156ed-873a-4485-a9a8-ff30725ea984","vmId":466} > for VM instance > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy","uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"} > > 2024-07-09 01:08:22,608 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) template 316 is already in store:6, type:Image > > > > From: Rohit Yadav <[email protected]> > > Date: Tuesday, July 9, 2024 at 12:44 AM > > To: [email protected] <[email protected]> > > Subject: Re: Unable to create new VMs or Console Proxy > > WARNING: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > > > Hi Marty, > > > > Can you check and ensure all your CloudStack hosts (management server, > kvm hosts if applicable and ssvm, cpvm) are on the same version (4.19.0.2) > & restarted post update/upgrade. > > > > Once you've done this, can you share your management server logs where > you see the exceptions/errors around VM deployment. You may also want to > check if you've any capacity issues, or using any offering that's tagged > (host or storage tags) for which you don't have available capacity? > > > > > > > > Regards. > > > > > > > > > > ________________________________ > > From: Marty Godsey <[email protected]> > > Sent: Tuesday, July 9, 2024 07:29 > > To: [email protected] <[email protected]> > > Subject: Re: Unable to create new VMs or Console Proxy > > > > Ok, figured, out the service offerings issues, I rebooted the database > and it came back. I still can not create VMs. > > > > It is stating that the storage is suitable, but it is in avoid set. I > have no idea why it is in avoid set since all hosts and the Cloudstack > management server can get to it. > > > > Where would I check this? > > > > From: Marty Godsey <[email protected]> > > Date: Monday, July 8, 2024 at 9:19 PM > > To: [email protected] <[email protected]> > > Subject: Unable to create new VMs or Console Proxy > > WARNING: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > > > Question. All of a sudden, I cannot create new VMs, and the console > proxy also can’t be created. As I spent the past hour looking, I have found > that in the console, when I go to offerings for compute, disk, or system, I > get a 404, and it says it can’t find anything for domain -1. But when I > look in the database, I see the offerings. I took this opportunity to > update to 4.19.0.2 since I was already on 4.19.0.1. At first I thought it > was storage but it is not. Running VMS and secStorage and VRs are all > running fine. > > > > Where should I look from here? Nothing was changed for the root domain. >
