Thanks for sharing Sean. For #1, if the global setting is disabled we assume the infra no longer requires secured KVM hosts and legacy behaviour is used (connection is loosely secure one-sided but not by proper certificates which aren't checked). I think this can become an issue if libvirtd was secured and then later the global setting was changed - in such a case admin needs to fix the libvirtd conf (make it tcp, unsecured). Feel free to raise an issue on Github for #1, which may even be fixed by explaining what you've found in docs.
Regards. Regards, Rohit Yadav ________________________________ From: Sean Lair <sl...@ippathways.com> Sent: Tuesday, March 16, 2021 2:43:24 AM To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: RE: Secure Live Migration for KVM Quick update so no one spends any time looking into this. Found a few things that we are working to fix: 1. if ca.plugin.root.auth.strictness is set to false, CloudStack will not try to renew any certs it has issued. I'd say this is an issue, it should still renew certs it has issued. 2. if the KVM agent is connecting to the management servers via a load-balancer, then the management servers see's the load-balancer's IP address as the client IP address. This causes the client certificate trust check to fail, as the load-balancer IP address is not in the cert's Subject Alternative Names list. 3. Similar to #2, the CA background task also has an issue when KVM agents come through a load-balancer We'll fix #2 and #3 by having the KVM agents connect directly to the mgmt servers. Thanks Sean rohit.ya...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue -----Original Message----- From: Sean Lair <sl...@ippathways.com> Sent: Monday, March 15, 2021 12:18 PM To: dev@cloudstack.apache.org Subject: RE: Secure Live Migration for KVM Hi Rohit from our initial debugging, the issue may be a little more involved. Maybe you could add some insight. We added some debug logging to monitor the size of the activeCertMap and have noticed it is almost always 0. When the CABackgroundTask runs, it never does anything because the in memory activeCertMaps on each mgmt server is empty. When a KVM host connects to a mgmt server, we do not see any code that populates the activeCertMap with the newly connected host's Cert. Shouldn't a host connection trigger adding the host's cert to the activeCertMap? Furthermore, when a cert is provisioned from the web-interface/API for a host, we do see the activeCertMap initially being populated. However, as part of that process, the agent is restarted. That restart of the agent triggers the following method in AgentManagerImpl.java: protected boolean handleDisconnectWithoutInvestigation(final AgentAttache attache, final Status.Event event, final boolean transitState, final boolean removeAgent) That method ends up calling the following method which removes the host/cert from the activeCertMap: caService.purgeHostCertificate(host); Now, since at host reconnect there isn't any code to re-populate the activeCertMap, it remains at 0 and as mentioned the CABackgroundTask never has anything to do, thus certs never get renewed. We are still looking into this, but let us know what we are missing if you have a chance to take a look. Thanks!! Sean -----Original Message----- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: Friday, March 12, 2021 12:50 AM To: dev@cloudstack.apache.org Subject: [DKIM Fail] Re: Secure Live Migration for KVM Hi Greg, I think you're right the https://github.com/apache/cloudstack/pull/4156 should fix the auto-renewal issue. In the meanwhile for already connected kvm hosts/agents, you can run the provisionCertificate API. Regards. ________________________________ From: Greg Goodrich <ggoodr...@ippathways.com> Sent: Friday, March 12, 2021 04:00 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: Secure Live Migration for KVM Further investigation finds this PR which may be related - https://github.com/apache/cloudstack/pull/4156. We are investigating if this could be the cause. -- Greg Goodrich | IP Pathways Development Manager 3600 109th Street | Urbandale, IA 50322 p. 515.422.9346 | e. ggoodr...@ippathways.com<mailto:j...@ippathways.com> rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue On Mar 11, 2021, at 4:09 PM, Greg Goodrich <ggoodr...@ippathways.com<mailto:ggoodr...@ippathways.com>> wrote: We have just discovered in our Lab environment that the certificates for libvirtd did not auto renew. Thus when we did an update, and restart of the agent, it failed to start, due to Libvirtd failing to start from an expired certificate. We then checked our production hosts, and their certificates are due to expire in 4 days, even though our setting is to auto renew at 15 days. Has anyone else encountered a problem with this? It appears to be related to this feature - https://github.com/apache/cloudstack/pull/2505. We are running 4.11.3 in both environments. -- Greg Goodrich | IP Pathways Development Manager 3600 109th Street | Urbandale, IA 50322 p. 515.422.9346 | e. ggoodr...@ippathways.com<mailto:ggoodr...@ippathways.com><mailto:j...@ippathways.com>
