Kubernetes stuck in the starting state
Hello All Kubernetes cluster is stucked in the 'Starting' state, even though VM's, LB, Firewall Rules are been created properly. Have any one faced such issue? [image: image.png] Any help would be really appreciated. Thank you! Regards, Sanjay Kumar
Re: Getting DB error while installing fresh cloudstack-mangment on Rocky-Linux9
Hi All, I was installing fresh management server of new release apache cloudstack 4.18 and getting db error. When I was installing apache cloudstack 4.17 then it is working but apache cloudstack 4.18 is not working. we are suspecting, there could be db issue or version issue? Any help will be greatly appreciated. On Thu, Mar 23, 2023 at 9:06 AM Sanjay Kumar wrote: > Hi Wei, > > Thanks for your prompt response > > Please find the logs herewith. > > > > On Wed, Mar 22, 2023 at 1:07 PM Wei ZHOU wrote: > >> Hi, >> >> Can you upload the full log ? >> >> -Wei >> >> On Wed, 22 Mar 2023 at 03:47, Sanjay Kumar wrote: >> >> > Hi All, >> > >> > I was trying to fresh installation of cloudstack 4.18 on rocky linux 9 >> and >> > getting db error. >> > >> > Can't write; duplicate key in table '#sql-3bb_2870 >> > >> > 2023-03-22 03:28:24,031 DEBUG [c.c.u.d.T.Transaction] (main:null) >> (logid:) >> > Rolling back the transaction: Time = 354 Name = Upgrade; called by >> > >> > >> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-DatabaseUpgradeChecker.upgrade:325-DatabaseUpgradeChecker.check:401-CloudStackExtendedLifeCycle.checkIntegrity:64-CloudStackExtendedLifeCycle.start:54-DefaultLifecycleProcessor.doStart:178-DefaultLifecycleProcessor.access$200:54-DefaultLifecycleProcessor$LifecycleGroup.start:356-Iterable.forEach:75-DefaultLifecycleProcessor.startBeans:155 >> > >> > 1. ACS_Version: 4.18 >> > 2. MysqlDB: 5.7.41 >> > >> > Any help would be really appreciated. Thank you! >> > >> > >> > With Regards, >> > >> > Sanjay >> > >> >
We are facing strange issue with acs 4.18 with Rocky Linux9
Hi All, We have setup the lab acs 4.18 with Rocky linux 9 and it was running but after a day we faced the issue. is there any OS dependencies with 4.18? Could not add host at [http://10.40.40.71] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.40.40.71 - Failed to setup keystore on the KVM host: 10.40.40.71]. 2023-03-27 10:09:11,382 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-3383eeb3) (logid:c03fe0a7) Detected management node left, id:2, nodeIP:10.40.40.23 2023-03-27 10:09:11,382 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-3383eeb3) (logid:c03fe0a7) Trying to connect to 10.40.40.23 2023-03-27 10:09:11,383 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-3383eeb3) (logid:c03fe0a7) Management node 2 is detected inactive by timestamp but is pingable 2023-03-27 10:09:12,882 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee783d0b) (logid:87ac62d2) Detected management node left, id:2, nodeIP:10.40.40.23 2023-03-27 10:09:12,882 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee783d0b) (logid:87ac62d2) Trying to connect to 10.40.40.23 2023-03-27 10:09:12,882 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee783d0b) (logid:87ac62d2) Management node 2 is detected inactive by timestamp but is pingable 2023-03-27 10:09:14,382 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Detected management node left, id:2, nodeIP:10.40.40.23 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Trying to connect to 10.40.40.23 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Management node 2 is detected inactive by timestamp but is pingable 2023-03-27 10:09:15,122 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-b26dd505) (logid:66b6c0db) HA health check task is running... [root@ASCLDACS01 ~]# tail /var/log/cloudstack/management/management-server.log 2023-03-27 10:09:14,382 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Detected management node left, id:2, nodeIP:10.40.40.23 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Trying to connect to 10.40.40.23 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Management node 2 is detected inactive by timestamp but is pingable 2023-03-27 10:09:15,122 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-b26dd505) (logid:66b6c0db) HA health check task is running... 2023-03-27 10:09:15,881 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3ca8d68) (logid:50c9cf85) Detected management node left, id:2, nodeIP:10.40.40.23 2023-03-27 10:09:15,881 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3ca8d68) (logid:50c9cf85) Trying to connect to 10.40.40.23 2023-03-27 10:09:15,882 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3ca8d68) (logid:50c9cf85) Management node 2 is detected inactive by timestamp but is pingable 2023-03-27 10:09:17,382 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2dbaab99) (logid:7f3a46b4) Detected management node left, id:2, nodeIP:10.40.40.23 2023-03-27 10:09:17,382 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2dbaab99) (logid:7f3a46b4) Trying to connect to 10.40.40.23 2023-03-27 10:09:17,383 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2dbaab99) (logid:7f3a46b4) Management node 2 is detected inactive by timestamp but is pingable Any help you can give would be greatly appreciated With Regards, Sanjay
Re: We are facing strange issue with acs 4.18 with Rocky Linux9
logs.zip <https://drive.google.com/file/d/1UrklbmUzr6xI_27v83zplHb2BpP9NBFO/view?usp=drivesdk> On Mon, Mar 27, 2023, 11:07 PM Wei ZHOU wrote: > Can you share the full log while added the host ? > > -Wei > > > > On Monday, 27 March 2023, Sanjay Kumar wrote: > > > Hi All, > > > > We have setup the lab acs 4.18 with Rocky linux 9 and it was running but > > after a day we faced the issue. is there any OS dependencies with 4.18? > > > > > > > > Could not add host at [http://10.40.40.71] with zone [1], pod [1] and > > cluster [1] due to: [ can't setup agent, due to > > com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore > > on the KVM host: 10.40.40.71 - Failed to setup keystore on the KVM host: > > 10.40.40.71]. > > > > > > 2023-03-27 10:09:11,382 DEBUG [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-3383eeb3) (logid:c03fe0a7) Detected management > > node left, id:2, nodeIP:10.40.40.23 > > 2023-03-27 10:09:11,382 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-3383eeb3) (logid:c03fe0a7) Trying to connect to > > 10.40.40.23 > > 2023-03-27 10:09:11,383 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-3383eeb3) (logid:c03fe0a7) Management node 2 is > > detected inactive by timestamp but is pingable > > 2023-03-27 10:09:12,882 DEBUG [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-ee783d0b) (logid:87ac62d2) Detected management > > node left, id:2, nodeIP:10.40.40.23 > > 2023-03-27 10:09:12,882 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-ee783d0b) (logid:87ac62d2) Trying to connect to > > 10.40.40.23 > > 2023-03-27 10:09:12,882 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-ee783d0b) (logid:87ac62d2) Management node 2 is > > detected inactive by timestamp but is pingable > > 2023-03-27 10:09:14,382 DEBUG [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Detected management > > node left, id:2, nodeIP:10.40.40.23 > > 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Trying to connect to > > 10.40.40.23 > > 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Management node 2 is > > detected inactive by timestamp but is pingable > > 2023-03-27 10:09:15,122 DEBUG [o.a.c.h.HAManagerImpl] > > (BackgroundTaskPollManager-6:ctx-b26dd505) (logid:66b6c0db) HA health > > check > > task is running... > > [root@ASCLDACS01 ~]# tail > > /var/log/cloudstack/management/management-server.log > > 2023-03-27 10:09:14,382 DEBUG [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Detected management > > node left, id:2, nodeIP:10.40.40.23 > > 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Trying to connect to > > 10.40.40.23 > > 2023-03-27 10:09:14,383 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-1f50e525) (logid:a4ef06ad) Management node 2 is > > detected inactive by timestamp but is pingable > > 2023-03-27 10:09:15,122 DEBUG [o.a.c.h.HAManagerImpl] > > (BackgroundTaskPollManager-6:ctx-b26dd505) (logid:66b6c0db) HA health > > check > > task is running... > > 2023-03-27 10:09:15,881 DEBUG [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-b3ca8d68) (logid:50c9cf85) Detected management > > node left, id:2, nodeIP:10.40.40.23 > > 2023-03-27 10:09:15,881 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-b3ca8d68) (logid:50c9cf85) Trying to connect to > > 10.40.40.23 > > 2023-03-27 10:09:15,882 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-b3ca8d68) (logid:50c9cf85) Management node 2 is > > detected inactive by timestamp but is pingable > > 2023-03-27 10:09:17,382 DEBUG [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-2dbaab99) (logid:7f3a46b4) Detected management > > node left, id:2, nodeIP:10.40.40.23 > > 2023-03-27 10:09:17,382 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-2dbaab99) (logid:7f3a46b4) Trying to connect to > > 10.40.40.23 > > 2023-03-27 10:09:17,383 INFO [c.c.c.ClusterManagerImpl] > > (Cluster-Heartbeat-1:ctx-2dbaab99) (logid:7f3a46b4) Management node 2 is > > detected inactive by timestamp but is pingable > > > > > > Any help you can give would be greatly appreciated > > > > > > > > With Regards, > > > > Sanjay > > >
Kubernetes stuck in the starting state and gets error state after some time.
Hello All Kubernetes cluster is stuck in the 'Starting' state, and gets error state after some time even though VM's, LB, Firewall Rules are been created properly. cloudstack version: 4.18.0 kubernetes version : Kubernetes 1.23.3 Any help would be really appreciated. Regards, S Kumar
Router VM fails to boot with kernel panic when overprovisioinig is > 1
Hi All, When overprovisioning factors are > 1, the virtual router vm is not able to boot and ends up in a kernel panic. ACS_Version: 4.18.0 Storage: NFS Any help would be really appreciated. Thank you! With Regards, S Kumar
Guest OS Rhel 9 and centos 9 going to kernel panic mode.
Hello, We are facing strange issue with guest Os rhel 9 and centos 9 going to kernel panic mode while deploying the vm. ACS_Version: 4.18 KVM_Host: Rocky_Linux 8.8 Any help would be much appreciated Thanks, Sanjay Kumar
Issue in Setup KVM host with agent 4.18.1
Hi All, Does the ARM RL300 server Support Cloudstack? We are facing while installing the Cloudstack agent 4.18.1.0. Any help would be really appreciated. Thank you! Thank you Sanjay Kumar
Re: ISCSI WITH KVM
Hello All, Any help on this? ACS_Verson: 4.19.1.1 On Fri, Oct 11, 2024 at 12:48 AM Sanjay Kumar wrote: > > Hi > > We are trying to add iscsi with kvm in our infra with multiple host and it > is working with one host only. is there any guide line for iscsi use as > share storage with kvm. > > Migration failed. > Exception during migrate: org.libvirt.LibvirtException: Unsafe migration: > Migration without shared storage is unsafe > > Any help would be really appreciated. Thank you! > > Regards, > SK > > > > > > > > >
ISCSI WITH KVM
Hi We are trying to add iscsi with kvm in our infra with multiple host and it is working with one host only. is there any guide line for iscsi use as share storage with kvm. Migration failed. Exception during migrate: org.libvirt.LibvirtException: Unsafe migration: Migration without shared storage is unsafe Any help would be really appreciated. Thank you! Regards, SK
Re: Preferred storage pool (preferred.storage.pool)
Hi Wei, Thanks for the quick revert. We have to cluster in our setup, one is rbd and one is local. I want to set the uuid of primary storage of local in preferred.storage.pool from global setting then the vm will create in local or rbd pool? Thank you On Tue, Sep 17, 2024 at 1:13 PM Wei ZHOU wrote: > What's your question ? > > It is an account-level setting, you can set different values per account or > domain. > > -Wei > > On Tue, Sep 17, 2024 at 9:25 AM Sanjay Kumar > wrote: > > > Hello! > > > > Please let us know if we use this option(preferred.storage.pool), then > all > > VMs will use this pool which uuid we will use with this setting. > > > > > > Any help would be really appreciated. Thank you! > > > > Regards, > > SK > > >
Preferred storage pool (preferred.storage.pool)
Hello! Please let us know if we use this option(preferred.storage.pool), then all VMs will use this pool which uuid we will use with this setting. Any help would be really appreciated. Thank you! Regards, SK
Re: Best storage solution for public cloud with KVM
Hi Nux, Thanks for the quick response. Currently, we are using RBD with KVM but there is some performance issue with storage side and we are planing to host more than 2000 VMs that's why we are looking stable and reliable solutions for public cloud for KVM. Thank you! On Fri, Nov 29, 2024 at 4:22 AM Nux wrote: > Depends what you are trying to achieve! > > There are many options, most of them very good, including local storage. > > On 2024-11-28 19:04, Sanjay Kumar wrote: > > Hi All, > > > > We are looking for best solution of storage for public cloud with KVM. > > We > > need help from cloudstack family to find out best storage solutions. > > > > Any help would be really appreciated. Thank you! > > > > Thank you! > > > > Regards, > > SK >
Best storage solution for public cloud with KVM
Hi All, We are looking for best solution of storage for public cloud with KVM. We need help from cloudstack family to find out best storage solutions. Any help would be really appreciated. Thank you! Thank you! Regards, SK
VMware to KVM migration issue
Hi All, We are trying to migrate the vm from vmware to kvm using cloudstack feature and getting error. virt-v2v conversion of the ovf is failed VMware : 8.0 KVM: ubuntu 20 ACS: 4.19 Any help would be really appreciated. Thank you! Regards, SK
Re: Best storage solution for public cloud with KVM
Hi Alex, Thanks for response. If we go with nfs, is there any issue related to performance. Therefore, we are trying to host more then 2000 vms on. Thank you! On Mon, Dec 2, 2024 at 10:36 AM Alex Mattioli wrote: > +1 on what Nux said. > Personally I'd go NFS, then CEPH, then local storage as my options. > > Alex > > > > > -Original Message- > From: Nux > Sent: 29 November 2024 00:56 > To: us...@cloudstack.apache.org > Cc: dev@cloudstack.apache.org; Sanjay Kumar > Subject: Re: Best storage solution for public cloud with KVM > > Sanjay, > > Ok, so if you are looking for CEPH improvements then there are companies > that deal with it, such as 42on, Croit etc - I am sure there are more, > perhaps others can complete the list. > In terms of commercial offerings, if your thing is distributed and > hyperconverged then Storpool or Linstor comes to mind. > > Last but not least, if you are ok with not having fail over, VMs on local > storage (SSD, NVME) are the fastest, so you could offer several tiers of > storage I guess to meet most needs. > Additionally NFS can still be a great solution if you plan it right and > it's easy to do and has very mature support in Cloudstack, it will likely > be faster than CEPH in many circumstances. > > The above is very generic advice and you should do more research before > committing to anything. > > HTH > > On 2024-11-29 06:36, Sanjay Kumar wrote: > > Hi Nux, > > > > Thanks for the quick response. > > > > Currently, we are using RBD with KVM but there is some performance > > issue with storage side and we are planing to host more than 2000 VMs > > that's why we are looking stable and reliable solutions for public > > cloud for KVM. > > > > Thank you! > > > > > > On Fri, Nov 29, 2024 at 4:22 AM Nux wrote: > > > >> Depends what you are trying to achieve! > >> > >> There are many options, most of them very good, including local > >> storage. > >> > >> On 2024-11-28 19:04, Sanjay Kumar wrote: > >> > Hi All, > >> > > >> > We are looking for best solution of storage for public cloud with KVM. > >> > We > >> > need help from cloudstack family to find out best storage solutions. > >> > > >> > Any help would be really appreciated. Thank you! > >> > > >> > Thank you! > >> > > >> > Regards, > >> > SK > >> >
I want to upgrade the ACS from 4.18.1.1 to 4.20.0.0
Hi All, Currently, we are using acs 4.18.1.1 and planing to upgrade to 4.20.0.0 for support gpu's like Nvidia H100 and Nvidia L40s. How is the safest way to upgrade from 4.18.1.1 to 4.20.0.0 ? and is gpu's will work for Nvidia H100 and Nvidia L40s. Below are the details of currently running infra. ACS: 4.18.1.1 KVM: ubuntu 20 and rocky 8.0 Storage: ceph quancy. Any help would be really appreciated. Thank you!
Connectivity Issue with Virtual Routers in CloudStack.
Hello all, We have observed recurring issue that has been observed recently in the CloudStack environment we use for hosting virtual machines. Specifically, we have identified that certain Virtual Routers lose internet connectivity, which causes disruptions to services such as VPNs and other network-dependent functions. This issue was first noticed with the infrastructure of the customer1 vpc. For the customer2 vpc case, we resolved the issue by recreating the VPC along with the associated guest networks. 1. Connectivity Issues: Check IP Ranges and Netmasks: Connectivity Checks: We use ping, arping, and traceroute within the virtual router to test connectivity to other VMs and external networks. Virtual Router Diagnostics: we did run "Run Diagnostics" and "Get Diagnostics" features as well but not getting any result. 2. Virtual Router Issues: Force Stop and Restart: We have tried but no luck. Destroy and Recreate: We have tried but no luck. DHCP Problems: There is no issue with dhcp. 3. Network Configuration: Network connection also ok. Any help would be really appreciated. Thank you! Regars, SK
Re: Connectivity Issue with Virtual Routers in CloudStack.
Hi Team, Please help on connectivity issue of virtual router of cloudstack. On Thu, Apr 17, 2025 at 11:21 AM Sanjay Kumar wrote: > Hello all, > > We have observed recurring issue that has been observed recently in the > CloudStack environment we use for hosting virtual machines. > > Specifically, we have identified that certain Virtual Routers lose > internet connectivity, which causes disruptions to services such as VPNs > and other network-dependent functions. This issue was first noticed with > the infrastructure of the customer1 vpc. For the customer2 vpc case, we > resolved the issue by recreating the VPC along with the associated guest > networks. > > 1. Connectivity Issues: > > Check IP Ranges and Netmasks: > > Connectivity Checks: > We use ping, arping, and traceroute within the virtual router to test > connectivity to other VMs and external networks. > > Virtual Router Diagnostics: > we did run "Run Diagnostics" and "Get Diagnostics" features as well but > not getting any result. > > 2. Virtual Router Issues: > > Force Stop and Restart: > > We have tried but no luck. > > Destroy and Recreate: > > We have tried but no luck. > > DHCP Problems: > > There is no issue with dhcp. > > 3. Network Configuration: > > Network connection also ok. > > Any help would be really appreciated. Thank you! > > > Regars, > SK > > > > > > > > >
