Hi Rohit from our initial debugging, the issue may be a little more involved. Maybe you could add some insight.
We added some debug logging to monitor the size of the activeCertMap and have noticed it is almost always 0. When the CABackgroundTask runs, it never does anything because the in memory activeCertMaps on each mgmt server is empty. When a KVM host connects to a mgmt server, we do not see any code that populates the activeCertMap with the newly connected host's Cert. Shouldn't a host connection trigger adding the host's cert to the activeCertMap? Furthermore, when a cert is provisioned from the web-interface/API for a host, we do see the activeCertMap initially being populated. However, as part of that process, the agent is restarted. That restart of the agent triggers the following method in AgentManagerImpl.java: protected boolean handleDisconnectWithoutInvestigation(final AgentAttache attache, final Status.Event event, final boolean transitState, final boolean removeAgent) That method ends up calling the following method which removes the host/cert from the activeCertMap: caService.purgeHostCertificate(host); Now, since at host reconnect there isn't any code to re-populate the activeCertMap, it remains at 0 and as mentioned the CABackgroundTask never has anything to do, thus certs never get renewed. We are still looking into this, but let us know what we are missing if you have a chance to take a look. Thanks!! Sean -----Original Message----- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: Friday, March 12, 2021 12:50 AM To: dev@cloudstack.apache.org Subject: [DKIM Fail] Re: Secure Live Migration for KVM Hi Greg, I think you're right the https://github.com/apache/cloudstack/pull/4156 should fix the auto-renewal issue. In the meanwhile for already connected kvm hosts/agents, you can run the provisionCertificate API. Regards. ________________________________ From: Greg Goodrich <ggoodr...@ippathways.com> Sent: Friday, March 12, 2021 04:00 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: Secure Live Migration for KVM Further investigation finds this PR which may be related - https://github.com/apache/cloudstack/pull/4156. We are investigating if this could be the cause. -- Greg Goodrich | IP Pathways Development Manager 3600 109th Street | Urbandale, IA 50322 p. 515.422.9346 | e. ggoodr...@ippathways.com<mailto:j...@ippathways.com> rohit.ya...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue On Mar 11, 2021, at 4:09 PM, Greg Goodrich <ggoodr...@ippathways.com<mailto:ggoodr...@ippathways.com>> wrote: We have just discovered in our Lab environment that the certificates for libvirtd did not auto renew. Thus when we did an update, and restart of the agent, it failed to start, due to Libvirtd failing to start from an expired certificate. We then checked our production hosts, and their certificates are due to expire in 4 days, even though our setting is to auto renew at 15 days. Has anyone else encountered a problem with this? It appears to be related to this feature - https://github.com/apache/cloudstack/pull/2505. We are running 4.11.3 in both environments. -- Greg Goodrich | IP Pathways Development Manager 3600 109th Street | Urbandale, IA 50322 p. 515.422.9346 | e. ggoodr...@ippathways.com<mailto:ggoodr...@ippathways.com><mailto:j...@ippathways.com>
