[+cc Tom, Brijesh] On 25/11/2021 15:42, Daniel P. Berrangé wrote: > On Thu, Nov 25, 2021 at 02:44:51PM +0200, Dov Murik wrote: >> [+cc jejb, tobin, jim, hubertus] >> >> >> On 25/11/2021 9:14, Sergio Lopez wrote: >>> On Wed, Nov 24, 2021 at 06:29:07PM +0000, Dr. David Alan Gilbert wrote: >>>> * Daniel P. Berrangé (berra...@redhat.com) wrote: >>>>> On Wed, Nov 24, 2021 at 11:34:16AM -0500, Tyler Fanelli wrote: >>>>>> Hi, >>>>>> >>>>>> We recently discussed a way for remote SEV guest attestation through >>>>>> QEMU. >>>>>> My initial approach was to get data needed for attestation through >>>>>> different >>>>>> QMP commands (all of which are already available, so no changes required >>>>>> there), deriving hashes and certificate data; and collecting all of this >>>>>> into a new QMP struct (SevLaunchStart, which would include the VM's >>>>>> policy, >>>>>> secret, and GPA) which would need to be upstreamed into QEMU. Once this >>>>>> is >>>>>> provided, QEMU would then need to have support for attestation before a >>>>>> VM >>>>>> is started. Upon speaking to Dave about this proposal, he mentioned that >>>>>> this may not be the best approach, as some situations would render the >>>>>> attestation unavailable, such as the instance where a VM is running in a >>>>>> cloud, and a guest owner would like to perform attestation via QMP (a >>>>>> likely >>>>>> scenario), yet a cloud provider cannot simply let anyone pass arbitrary >>>>>> QMP >>>>>> commands, as this could be an issue. >>>>> >>>>> As a general point, QMP is a low level QEMU implementation detail, >>>>> which is generally expected to be consumed exclusively on the host >>>>> by a privileged mgmt layer, which will in turn expose its own higher >>>>> level APIs to users or other apps. I would not expect to see QMP >>>>> exposed to anything outside of the privileged host layer. >>>>> >>>>> We also use the QAPI protocol for QEMU guest agent commmunication, >>>>> however, that is a distinct service from QMP on the host. It shares >>>>> most infra with QMP but has a completely diffent command set. On the >>>>> host it is not consumed inside QEMU, but instead consumed by a >>>>> mgmt app like libvirt. >>>>> >>>>>> So I ask, does anyone involved in QEMU's SEV implementation have any >>>>>> input >>>>>> on a quality way to perform guest attestation? If so, I'd be interested. >>>>> >>>>> I think what's missing is some clearer illustrations of how this >>>>> feature is expected to be consumed in some real world application >>>>> and the use cases we're trying to solve. >>>>> >>>>> I'd like to understand how it should fit in with common libvirt >>>>> applications across the different virtualization management >>>>> scenarios - eg virsh (command line), virt-manger (local desktop >>>>> GUI), cockpit (single host web mgmt), OpenStack (cloud mgmt), etc. >>>>> And of course any non-traditional virt use cases that might be >>>>> relevant such as Kata. >>>> >>>> That's still not that clear; I know Alice and Sergio have some ideas >>>> (cc'd). >>>> There's also some standardisation efforts (e.g. >>>> https://www.potaroo.net/ietf/html/ids-wg-rats.html >>>> and https://www.ietf.org/archive/id/draft-ietf-rats-architecture-00.html >>>> ) - that I can't claim to fully understand. >>>> However, there are some themes that are emerging: >>>> >>>> a) One use is to only allow a VM to access some private data once we >>>> prove it's the VM we expect running in a secure/confidential system >>>> b) (a) normally involves requesting some proof from the VM and then >>>> providing it some confidential data/a key if it's OK >>>> c) RATs splits the problem up: >>>> >>>> https://www.ietf.org/archive/id/draft-ietf-rats-architecture-00.html#name-architectural-overview >>>> I don't fully understand the split yet, but in principal there are >>>> at least a few different things: >>>> >>>> d) The comms layer >>>> e) Something that validates the attestation message (i.e. the >>>> signatures are valid, the hashes all add up etc) >>>> f) Something that knows what hashes to expect (i.e. oh that's a RHEL >>>> 8.4 kernel, or that's a valid kernel command line) >>>> g) Something that holds some secrets that can be handed out if e & f >>>> are happy. >>>> >>>> There have also been proposals (e.g. Intel HTTPA) for an attestable >>>> connection after a VM is running; that's probably quite different from >>>> (g) but still involves (e) & (f). >>>> >>>> In the simpler setups d,e,f,g probably live in one place; but it's not >>>> clear where they live - for example one scenario says that your cloud >>>> management layer holds some of them, another says you don't trust your >>>> cloud management layer and you keep them separate. >>>> >>>> So I think all we're actually interested in at the moment, is (d) and >>>> (e) and the way for (g) to get the secret back to the guest. >>>> >>>> Unfortunately the comms and the contents of them varies heavily with >>>> technology; in some you're talking to the qemu/hypervisor (SEV/SEV-ES) >>>> while in some you're talking to the guest after boot (SEV-SNP/TDX maybe >>>> SEV-ES in some cases). >> >> SEV-ES has pre-launch measurement and secret injection, just like SEV >> (except that the measurement includes the initial states of all vcpus, >> that is, their VMSAs. BTW that means that in order to calculate the >> measurement the Attestation Server must know exactly how many vcpus are >> in the VM). > > Does that work with CPU hotplug ? ie cold boot with -smp 4,maxcpus=8 > and some time later try to enable the extra 4 cpus at runtime ? >
AFAIK all generations of SEV don't support CPU hotplug. Tom, Brijesh - is that indeed the case? I don't know about TDX. -Dov > >>>> So my expectation at the moment is libvirt needs to provide a transport >>>> layer for the comms, to enable an external validator to retrieve the >>>> measurements from the guest/hypervisor and provide data back if >>>> necessary. Once this shakes out a bit, we might want libvirt to be >>>> able to invoke the validator; however I expect (f) and (g) to be much >>>> more complex things that don't feel like they belong in libvirt. >>> >>> We experimented with the attestation flow quite a bit while working on >>> SEV-ES support for libkrun-tee. One important aspect we noticed quite >>> early, is that there's more data that's needed to be exchange of top >>> of the attestation itself. >>> >>> For instance, even before you start the VM, the management layer in >>> charge of coordinating the confidential VM launch needs to obtain the >>> Virtualization TEE capabilities of the Host (SEV-ES vs. SEV-SNP >>> vs. TDX) and the platform version, to know which features are >>> available and whether that host is a candidate for running the VM at >>> all. >>> >>> With that information, the mgmt layer can build a guest policy (this >>> is SEV's terminology, but I guess we'll have something similar in >>> TDX) and feed it to component launching the VMM (libvirt, in this >>> case). >>> >>> For SEV-SNP, this is pretty much the end of the story, because the >>> attestation exchange is driven by an agent inside the guest. Well, >>> there's also the need to have in the VM a well-known vNIC bridged to a >>> network that's routed to the Attestation Server, that everyone seems >>> to consider a given, but to me, from a CSP perspective, looks like >>> quite a headache. In fact, I'd go as far as to suggest this >>> communication should happen through an alternative channel, such as >>> vsock, having a proxy on the Host, but I guess that depends on the CSP >>> infrastructure. >> >> If we have an alternative channel (vsock?) and a proxy on the host, >> maybe we can share parts of the solution between SEV and SNP. >> >> >>> For SEV/SEV-ES, as the attestation happens at the VMM level, there's >>> still the need to have some interactions with it. As Tyler pointed >>> out, we basically need to retrieve the measurement and, if valid, >>> inject the secret. If the measurement isn't valid, the VM must be shut >>> down immediately. >>> >>> In libkrun-tee, this operation is driven by the VMM in libkrun, which >>> contacts the Attestation Server with the measurement and receives the >>> secret in exchange. I guess for QEMU/libvirt we expect this to be >>> driven by the upper management layer through a delegated component in >>> the Host, such as NOVA. In this case, NOVA would need to: >>> >>> - Based on the upper management layer info and the Host properties, >>> generate a guest policy and use it while generating the compute >>> instance XML. >>> >>> - Ask libvirt to launch the VM. >> >> Launch the VM with -S (suspended; so it doesn't actually begin running >> guest instructions). >> >> >>> >>> - Wait for the VM to be in SEV_STATE_LAUNCH_SECRET state *. >>> >>> - Retrieve the measurement *. >> >> Note that libvirt holds the QMP socket to QEMU. So whoever fetches the >> measurement needs either (a) to ask libvirt to it; or (b) to connect to >> another QMP listening socket for getting the measurement and injecting >> the secret. > > Libvirt would not be particularly happy with allowing (b) because it > enables the 3rd parties to change the VM state behind libvirt's back > in ways that can ultimately confuse its understanding of the state > of the VM. If there's some task that needs interaction with a QEMU > managed by libvirt, we need to expose suitable APIs in libvirt (if > they don't already exist). > > > Regards, > Daniel >
